Exchange Server / Outlook Message Formatting –> .MSG to .EML Conversion, Rich Text and TNEF (Transport Neutral Encapsulation Format)
2010-02-17 Update
2010-02-03 Initial Post
SUMMARY
Depending on the circumstances, Exchange will convert an attached e-mail (this means an existing e-mail that is attached/embedded into a new e-mail) from .MSG format to .EML format. The reason for the conversion has to do with the fact that Rich Text and TNEF formats are proprietary to MS Exchange/Outlook.
ISSUE
User said he used to be able to send attached e-mails (this means an existing e-mail that is attached/embedded into a new e-mail) to external recipients. Now the messages just seem to go into a black hole because the recipients never receive them and the sender never receives an NDR (non-delivery report). Regular messages go through fine; just the ones with attached e-mails have the issue. Note that attached e-mails in Outlook (tested with 2002, 2003, and 2007) automatically default to .MSG format.
After some investigation, I found that the attached e-mails were getting converted to .EML format somewhere during transport. Our e-mail security appliance blocks messages with .EML attachments and doesn’t notify the senders. Great, now we know what happened to those messages with attached e-mails.
If the user sent the attached e-mails as .MSG files, what is converting them to .EML before they get to the security appliance? All servers before the appliance are Exchange, so the conversion must be happening within Exchange. But looking in the user’s Sent Items, we do see that the attachments are still in .MSG format, so the conversion isn’t taking place in Outlook or on the user’s Exchange server, right? If that were the case, the sent e-mails in question would have .EML attachments, right?
What really puzzled me was that I was using a test computer that had Outlook 2003 and every time I sent an attached e-mail to my Gmail, it was delivered. But on my laptop, with Outlook 2007, and within Citrix using Outlook 2003, the attachments would get converted to .EML and get blocked by the security appliance.
One day I decided to send some test e-mails to Gmail with no attachments. I checked the headers in Gmail and noticed that the messages from the test computer were in TNEF format while the ones from my laptop and Citrix were not (see excerpts from headers below). Bingo! The TNEF-formatted messages kept the e-mail attachment in .MSG format while the non-TNEF messages converted the .MSG attachment to .EML.
After this finding, I forced Outlook on my laptop and in Citrix to send to Gmail in Rich Text (TNEF). Sure enough, the .MSG attachments came through fine. (Note that I used Outlook 2003 to access Gmail using IMAP, so I was able to see the Rich Text/TNEF messages intact). Mystery solved!
So after two weeks and talking to 3 support techs at MS, I found the solution myself. The pathetic thing is that I’ve known about the setting used to fix this issue for years, and so did the MS techs, but we just weren’t able to piece everything together. And the MS techs kept telling me that nothing in Exchange can convert an attachment (see notes contradicting this in the BACKROUND section), so I was at a lost. And the issue with the appliance blocking .EML files, and not sending NDRs to the sender, threw another wrench into the issue.
Excerpt from header of message sent from test computer (Rich Text is forced):
X-MS-TNEF-Correlator: <4658D0590BB64B43920E965453F69ECE0168118E@srv-ex-1.some-company.corp>
Excerpt from header of message sent from my laptop and Citrix (Let Outlook decide is selected):
X-MS-TNEF-Correlator:
RESOLUTION
For whatever reason, the user no longer had the selection to force Rich Text format when sending to the recipients. After changing the setting back, he was able to successfully send .MSG attachments to the recipients.
BACKGROUND
Note that in most cases, you don’t want to force Rich Text formatting because it’s proprietary to Exchange/Outlook. If you are sure that the recipients have Outlook and that their servers won’t convert the messages during transport, you can force Rich Text format to those particular recipients only.
Selecting to send in Rich Text format tells Exchange to send the message using TNEF. I kept asking the MS techs if there was a setting in Outlook that tells Exchange to convert certain attachments. They kept telling me that there wasn’t such a setting, which I have now proven their comment wrong.
Exchange normally won’t convert the actual attachments, but only encode them differently depending on particular settings. Based on my findings, Exchange will convert .MSG attachments. This seems logical and makes perfect sense because .MSG is a proprietary format for Exchange/Outlook. If Exchange didn’t convert the .MSG attachment, the recipient must have Outlook in order to open it. By converting the .MSG attachment to .EML (note that .EML is an RFC 822-compliant format), it allows any RFC 822 compliant e-mail client to open the file.
The important thing to understand here is that an .MSG file is an actual e-mail packaged into a file, so it’s not the same as an .XLS or .DOC attachment. Exchange is able to covert .MSG to .EML while retaining important information such as the header, message body, and attachments. With other file attachments, it makes no sense for Exchange to convert them. For example, Exchange wouldn’t be able to convert an .XLS file because Exchange doesn’t understand .XLS files—it only sees them as regular binary files.
And the reason that the message in the sent items still had the attachment in .MSG format is because the message itself never changed. Exchange converts the message to TNEF format on-the-fly during the transport/routing process, so the original message itself is never modified.
Another thing that I noticed is that, regardless of which format the message is delivered as, the recipient Outlook client will "reconstitute" the attached e-mail back to .MSG format even if it was in .EML format when received. We ended up allowing .EML files and I verified that our security appliance saw a forced Plain Text format e-mail with a .EML attachment leave our system, but once I received it, my Outlook 2003 client (connected to Gmail via IMAP), "reconstituted" the attachment back to .MSG format. If I selected to save the attachment, I saw that it's .MSG, so that's how I can tell. This is very interesting how all that's done in the background by Outlook.
ADDITIONAL RESOURCES
##### From http://office.microsoft.com/en-us/outlook/HA010347811033.aspx 2010-02-03:
To preserve RTF format for messages sent to a particular Internet recipient
- If you haven't already done so, add the recipient as a contact.
- With the contact open, in the E-mail box, double-click the contact's e-mail address.
- In the Internet Format list, select Send using Outlook Rich Text format.
Note If you create messages in plain text or HTML, this setting will also convert those messages to RTF when sent to the specified contact.
[If you’re sending to an external contact who isn't in your Outlook Contacts, you can just double-click on the e-mail address in the To field of the new message window and you’ll get the options to change the Internet Format.] #####
Below are some notes that I had written about TNEF, Rich Text, and Windmail.dat back in 2010-01-15. I’m going to include them here since it makes more sense to put all this info in a single post.
• Formatting settings are only used for messages leaving the Exchange org. Rich-text formatting is retained for all internal messages.
• TNEF (Transport Neutral Encapsulation Format) is basically a packaging format for the rich-text-formatted version of the message.
• Rich-text in Exchange/Outlook is proprietary to MS and is not the same as the rich-text format of word processing programs.
• Features that require rich-text/TNEF, such as voting buttons, require the external recipient to have Outlook also. But note that the TNEF data may be stripped from a message during transport, depending on the server settings. That would result in the recipient only getting a plain text version of the message with no formatting, voting buttons, etc., even if the recipient has Outlook.
• You can specify which external domains do or do not receive rich-text formatted messages by creating an entry for the domain in ESM --> Internet Message Formats.
• Users can also specify, in Outlook, the format of a message sent to external recipients.
IMPORTANT NOTE: The settings for Internet Format are not the same as the settings for the format of the message itself. This is confusing, but you can format a message as HTML and then force Rich Text for specific external recipients of the same message. Recipients that are in the same Exchange org as the sender’s will always receive messages in Rich Text/TNEF—even if the message itself is in plain text or HTML, the backend is always Rich Text/TNEF.
##### From “How e-mail message formats affect Internet e-mail messages in Outlook” at http://support.microsoft.com/kb/290809 on 2010-01-15:
A TNEF-encoded message contains a plain text version of the message, and a binary attachment that "packages" various other parts of the original message. In most cases, the binary attachment is named Winmail.dat, and may include the following information:
* The formatted text version of the message (for example, font information and colors).
* OLE objects (for example, embedded pictures and embedded Microsoft Office documents).
* Special Outlook features (for example, custom forms, voting buttons, and meeting requests).
* Regular file attachments that were added to the original message.
In addition to the previously listed information, the path to your personal folders (.pst) file and your log on name are embedded in the Winmail.dat file. Although this data is not explicitly exposed to the recipient, if the recipient opens the Winmail.dat file for editing in a binary or text editor, they can see the path and log on name. Note that the password information is not revealed. To ensure that the path to your .pst file or your log on name is not included in the Winmail.dat attachment, use the steps in this article to send messages that does not include the Winmail.dat file.
Some Outlook features require TNEF encoding to be understood correctly by an Internet e-mail recipient who also uses Outlook. For example, when you send a message with Voting buttons to a recipient over the Internet, if TNEF is not enabled for that recipient, the Voting buttons are not received. Alternatively, for sending messages with regular file attachments, TNEF is not needed. If you are sending messages with file attachments to a recipient who does not use Outlook or the Exchange Client, you should manually choose to use an e-mail format that does not require TNEF (such as plain text). By not sending TNEF messages, the recipient is able to view and save the attachments as expected. . .
In addition to the receiving client, it is not uncommon for an e-mail server to strip out TNEF information from messages as it delivers them. If a server option to remove TNEF is turned on, clients always receive a plain text version of the message. Exchange Server is an example of an e-mail server program that has the option to remove TNEF from messages. #####
Also see "How to configure Internet e-mail message formats at the user and the domain levels in Exchange Server 2003" at http://support.microsoft.com/kb/821750.
Tags: .EML, .MSG, Exchange Server 2003, HTML, MAPI, Outlook 2003, Outlook 2007, Plain Text, RFC 822, Rich Text, TNEF, Winmail.dat
