I eat tofu, & my test for my age was high normal (last time it was checked a while ago now). phytoestrogens may have a mild effect similar to oestrogen, they may have a slightly anti-oestrogen effect, studies vary (often depending on who's funding). Having an ordinary amount is not going to feminise you more than having any animal milk for example (see this link for the latest study showing cows milk does decrease testosterone)
). Also soya isn't the only food with phytoestrogens in them, many plant foods have them, so unless you avoid many foods, then you'll have some issues.
Basically, treat it like any food & have it on occasion.
Focus on unprocessed food (so more on beans, nuts, seed & whole grains) rather than fake meats (dieting for bodybuilding contests is a different matter, but for most people whole food is vastly superior). Try to make your food more 'natural' than buying processed stuff like burgers, etc.
Having the odd bit of tofu would do you any harm whereas cramming yourself with processed stuff everyday will.
With soya (like any food) don't go mad. If you eat tofu, don't also eat soya protein powder, soya milk, fake soya meat product etc. Drink another 'milk' (hemp, almond etc), make a legume salad (remember every bean is slightly different so buy a variety. Eat plenty of nuts & seeds, try a different protein powder.
Also if your cholesterol is fine & you're worried about testosterone levels try adding some coconut fat into your diet. A lot of larger guys & girls (both vegan & not) have found increased test by adding coconut fat the diet (I had a quick look for studies, just about all have been done on various animals of various species-cheapness I suppose, but I know many people who have had good effects using coconut fat). As a general note - before someone states "Ahh, so animal experiments discovered coconut fat raises testosterone"...well no actually the first reference I have to coconut fat was from Joseph Hise (the guy who popularised the back squat) back in the late 1920's-early 1930's, so yes it was a guy experimenting on himself & publishing the results in a physical culture magazine that is the first discoverer of the idea as far as I know! He may not have specified testosterone as the reason, but certainly advocated coconut fat for increased muscle size & libido (I'd have to check if testosterone was mentioned in the actual magazines).
The idea mainly disappeared with steroids as these drugs do bad things to the cholesterol levels so having them & coconut fat didn't mix well! Luckily the old information still survives in old magazines!
Hopefully that's been of some use.