PS3 Media Server absolutely transcodes mkvs. Standard would be MEncoder, but I use VLC instead.
I was curious about this to see why you were having issues and I wasn't. There is an option in PS3 Media Server that allows the user to remux or transcode files rather than autodetect. Some MKVs require autodetection to be turned off, and their contents trancoded. This is because the MKV container allows a greater spectrum of video codecs than the PS3 supports.
As of right now, I can stream an MKV file to the PS3 and it isn't being transcoded (I can tell because there is minimal CPU usage); however, when I try to play my copy of Avatar, it does transcode the MKV rather than remux. On the display, I can't see any reduction in quality and the audio is still 5.1 (haven't tested a 7.1 file yet). The transcoder uses only 4 cores with around 65% cpu usage (over those 4 only, less than 33% total) and is pushing out 1080p @ 42 fps. I'm using a dual-CPU Xeon X5492 @ 3.4Ghz (8 total cores).
I would think on a standard desktop computer, this would relate to around 50%+ CPU usage, which I think is fair considering what's being done.
And for why I don't use DTS? I don't have the money to go wasting on a theatre system that I can't install properly since I don't own my home. And since I don't have a DTS system, I can still get hit by Cinavia.
I don't own my home, and I still installed speakers, I just spoke to the owner first and they were actually excited that I agreed to leave the modifications in the wall after I left.
But again, I'm talking about "home theater." I can't really imagine a home theater without a sound system (to me, sound is more than 50% of the experience). If you're just casually watching then I'm not really talking about your particular situation. The PS3 is an invaluable tool in my setup, as a gaming platform, a sometimes media player, a better browser than what's offered on the TV (although my PC is hooked up), and most of all as a bluray player - which is it's primary function.
And as for Plex, what do I need an app for Plex on my Samsung TV, when the native player works absolutely fine, autoplays sequentially, gives me access to all audio streams/subtitles, and plays MKV natively.
I have used the Samsung 52" 240hz dual-core SmartTV extensively so I know what you're probably using at home. It's media player is very similar to what's in the LG SmartTV that I ultimately decided to get (due to it's better color production, deeper blacks, and passive 3D - but the Samsung is right up there).
Both TV's support most file formats natively, and
Plex does not transcode when sending media to either. What Plex lacks is the sequential playback, which sucks, but I would trade that for the features it brings including the ratings, box covers, folder management, autosync with torrents, etc. It's absolutely amazing, and has a very sleek interface that works wonderfully with 3D.
And as far as selecting A/V streams, you can do that with Plex, I dunno why you're saying you can't. Once the media starts, you can select whichever streams you want to play. I use Plex every single day, and I've never found a single media file that wouldn't play at 1080p w/3D and HD audio.
So why would I want to use an app to transcode the files for my TV when my TV plays them natively?
Plex won't transcode. The app is there to present the media in a beautiful carousel-like way with box art, reviews, information, and ratings from rotten tomatoes, metacritic, and IMDB. It also saves your place, autosyncs, and is more secure than using a Samba share. I really don't understand the reason not to use Plex - it doesn't transcode the files.