Updated HTML parser

Quick summary

The BEE editor will write HTML emails using a new technique that produces smaller messages. This should dramatically reduce the chances of "email clipping" in Gmail and Yahoo! Mail. Here is everything you need to know. If you have embedded BEE in your app, you will also find ways to test this change before it goes live in early March.

Updated HTML for BEE emails

On March 8, 2017, we released a major update to our HTML parser, which now produces state-of-the-art, thinner, better HTML.

When you use the editor, BEE saves your design into a JSON file. The HTML parser is the component that transforms that JSON file into ready-to-send HTML emails.

It's one of the services at the core of BEE, used by all BEE products (Free, Pro, and Plugin).

Post-release updates

This was a major release, so we are quickly fixing issues that are being reported (we conducted extensive BETA testing, but some issues did not surface during the BETA).

On March 31, 2017, we released a small patch for the new HTML parser that addressed the following, known issues:
  • Fixed issue with spacing in MS Outlook
  • Edited code to maintain button background color in certain email platforms that were not diplaying it

 

On March 16, 2017, we released a small patch for the new HTML parser that addressed the following, known issues:
  • Fixed issue with dividers (wrong width or position in Outlook)
  • Full-width image shown on mobile when physical width is larger than column
  • Fixed syntax issue in negative conditional statement (unclear if it caused issues)

 

On March 13, 2017, we released a small patch for the new HTML parser that addressed the following, known issues:
  • Lateral padding issue in Microsoft desktop clients
  • Resolution of issue in Microsfot Web clients (e.g. outlook.com, Office 365)
  • Images shown at 100% when width is bigger than column width on mobile clients
  • Typo in "min-width" declaration (unclear if it was causing rendering problems)

 

Why this update: lighter, better HTML

One of the major reasons people use BEE for designing emails is the peace of mind that it provides with regard to how those messages will render in the inbox.

Gmail, as one of the most commonly used email clients, is one of the priorities for our team. That includes ensuring responsive behavior in the Gmail mobile app.

To achieve responsiveness in the Gmail app and other mobile applications, we adopted the fluid hybrid design technique (more on this).

Although this worked great in most cases, it did present a drawback: verbose HTML code that resulted in bigger HTML documents. 

While the HTML used in email contains many nested HTML tables, the hybrid method introduced additional code, doubling the total message size in some cases. As both Gmail and Yahoo! Mail "clip" email messages that are larger than 102KB in size, this caused the content of some of the messages to be partially hidden in the inbox.

To give you an idea of what we are referring to, here is how a one column row with a simple text block was written in HTML using the fluid hybrid design technique. A total of 63 lines of lengthy code. Here are just the first 30... (click here to view the full HTML).

Easier and lighter code: modern HTML comes to the rescue

Until recently, email developers had to choose between coding messages that may be clipped by Gmail and Yahoo, and emails that are not responsive in the Gmail App, the most used email client on Android-powered phones.

As a result of this dilemma, many talented developers started looking for new techniques. 

In particular, inspired by Rémi Parmentier's Fab Four technique, email coders were able to develop a new model that uses DIVs instead of nested tables to create fluid layouts, dramatically simplifying the final output.

The reduction in the HTML footprint is remarkable. Look at the same content mentioned above, rewritten using this new technique by BEE's new HTML parser (click here to view the full HTML).

Here is how the code has changed:

  • 17 lines of code vs. 63.
  • 2KB vs. 5KB (when saved as an HTML file).
  • A much simpler HTML markup (*)

(*) Still too many HTML tags? Well, the HTML that BEE outputs is not hand-coded, but rather a structure that must work in many scenarios with any content that you may add, so we need a lot of fallback elements (smile) 

Adding support for Microsoft email clients

Unfortunately, the code above cannot be used "as is" in Microsoft Outlook desktop or Windows Mail. These applications don't use an Internet browser to render the HTML, but rather the MS Word rendering engine, forcing the introduction of some additional code.

Specifically:

  • Conditional statements to apply the correct BODY CSS reset.
  • Conditional statements to add tables around content blocks.
  • Negative conditional statements to hide some content.
  • VML macros to build beautifully rounded buttons.

With this additional code in place, our sample HTML snippet becomes a bit more complex, but still much smaller than the original (click here to view the full HTML).

Improvements summary

Here is a summary of the main benefits introduced by using these new HTML coding techniques:

  • File size reduction (about 40% less HTML code for the same results).
  • Message clipping much more rare thanks to the reduced file size.
  • Faster delivery time thanks to the smaller file size.
  • Buttons that are 100% clickable (the entire button, not just the text in the button).
  • Display improvements in the most challenging email clients (IBM Notes, Outlook 2000/03).

Feature support changes

There are a few drawbacks, but they are far outweighed by the benefits. Drawbacks include:

  • Background images don't display in the Outlook.com email client.
  • There are multi-column display problems in some less popular webmail clients:
    • Comcast
    • GMX.de
    • Web.de

Sample HTML files for testing

These are some of our BEE Pro templates turned into HTML emails with the new parser:

You can use the sample HTML files listed above in your application to test sending some messages and confirm that there are no compatibility issues. Specifically, this is what you should be looking for:

  • There are no differences between the received HTML and the starting code
  • Additions (merge contents or other placeholder replacements) are correctly confined in the given container element
  • Additions are not modifying the message layout

Please note that some applications split HTML messages into separate parts to facilitate code post-processing. In this scenarios, the final HTML is assembled during the sending process.

If this is your case, ensure that the following, new conditional elements do not create an issue. In particular, check that the following is true when the HTML is saved and sent by your application:

1. The HTML tag is wrapped with an Internet Explorer (IE) conditional structure. Get the code.

2. The BODY tag is wrapped with a Microsoft Outlook conditional structure. Get the code. Here is a preview:

Testing the new HTML parser

Would you like to test the new HTML parser with your application messages?

To help you with this, we published a public endpoint that can be contacted with Postman.

Download and import the Postman collection to start using it or apply the following settings:

  • Request: POST
  • Endpoint: https://parser-beta.getbee.io/api/nlparser?ispreview=false
  • Body: message JSON with the options raw / JSON (application/json)

The response body should contain the HTML created for the provided JSON.

If you are testing the reduction in the size of the HTML file, please consider that Postman's pretty version of the HTML will be larger due to the formatting applied to it. Use the Raw response instead to get the true message size.

Have more questions? Submit a request

Comments

  • Avatar
    kent

    Hi Massimo, has much testing been performed using the Apple Mail Mac and iOS clients? I plan to test these myself, however, the links to 4 the samples above are returning an authentication error. Thanks

  • Avatar
    Guille Padilla

    Hi Kent, sorry for the broken links, we'd some problems to share files inside the article, so moved them externally. All the links may be working now.
    We perform tests with testing tools (Litmus, Email On Acid) and directly with the most used or most problematic clients. Fortunately, Apple mail and iOS mail use an advanced rendering engine, doing things simpler for email coders.
    Please, let us know if you find any inconsistence across your targeted clients.

  • Avatar
    muhan95

    Hi, we've just started testing this and it works perfectly on internet browser rendering clients however we've encountered many issues on outlook. The final HTML renders very differently, completely breaking our layouts.

    Please could you have a look into this before going live with this update on March the 3rd as I'm sure many others will be affected by it.

     

    Edit: After further testing, we're pretty sure there's something wrong with the non-mso conditional statement for rounded-corner buttons or something related to that section of the code.

    Edited by muhan95
  • Avatar
    Guille Padilla

    Hi Muhan, we tested it in different Outlook versions without issues. Please, send to support at beefree.io one or more of the JSON used in your tests to check them.
    If there is something that doesn't work as expected, we will fix it ASAP.

    Edited by Guille Padilla
  • Avatar
    gharmel

    ***EDIT- This issue below seems to be due to improperly encoded json***

    My first simple test of the new process caused me some concern. I am hopefully doing something wrong.

    JSON used for both tests: https://dar.uga.edu/apps_dev/htmlemails/viewjson/10272

    HTML output for existing parser: https://dar.uga.edu/apps_dev/htmlemails/viewjson/10272

     

    Issues noticed so far:
    -Spacing in text block changed between paragraphs
    -Button Size and Font Changed
    -Caption location and font changed
    -Most links didn't render and were removed (see the bottom line of the email). -This might be due to the fact that we need irregular link formats like "target=&pid=393&did=0&tab=0" in order for our emailing system to correctly process them.

    Edited by gharmel
  • Avatar
    Guille Padilla

    Hi gharmel, could you please directly link the JSON files?
    If it's not possible, please, send them to support at beefree.io.

  • Avatar
    gharmel

    This is what I used for both tests: https://dar.uga.edu/apps_dev/htmlemails/viewjson/10272

  • Avatar
    kent

    Thanks Guille, I've tested a couple of our templates and the only thing that stands out is that the width of the buttons has increased with the new parser. No other differences to report at this stage.

    Will there be an option to remain on the old parser if for some reason we need to? Eg, via a setting when setting up the editor?

  • Avatar
    Guille Padilla

    Hi gharmel, this is an HTML file. We can't test it, only a validated JSON can be sent to the parser.

  • Avatar
    Guille Padilla

    Hi Kent, some issues with buttons had been corrected. With the table based markdown there was a width difference between the standard button and the VML button. I'm not sure if this can cause what you comment, I'll check it.
    The parser will not be replaced but updated, so there won't be an option to switch between the running version and an old version. If your application have problems to use the new markup, please, write us at support at beefree.io with your case. We will do our best to give you a solution.

  • Avatar
    gharmel

    Guille,

    You helped me find what I was doing wrong. The link I gave didn't have the application type in the header of application/json and it wasn't utf-8 encoded...but the postman process still processed it (thus my confusion).

    I fixed how I output the json and retried it, and the issues I mentioned were resolved.

  • Avatar
    Guille Padilla

    Ey gharmel, thanks for the update.

  • Avatar
    kent

    Thanks Guille, buttons are looking good now. I don't think there will be any concerns with the change over to the new parser.

  • Avatar
    Nishant Bafna

    Hi,

    We have implemented BEE email editor in our application (the legacy parser).

    Is the new parser available via the new endpoint only or will that be available via the old endpoint as well?

    Thanks,
    Nishant

  • Avatar
    gharmel

    Now that the issues have been addressed, do we need to redo our testing, or will the rendered html be the same as when you first announced the new parser?

  • Avatar
    Simon Mahler

    Hi Bees, what does the parameter ?ispreview=false at the testing endpoint mean?
    I set it to true and found out, that the generated HTML works much better in GMX.de and Web.de.
    Is there any hope, that the mentioned layout problems with these providers are solved soon?

  • Avatar
    Massimo Arrigoni

    To answer some of the recent questions (unfortunately ZenDesk does not allow for threaded comments):

    Nishant: the current parser used by the BEE editor will switch automatically to the new parser. In other words, any implementation of BEE Plugin will start outputting HTML generated with the new parser (i.e. any method that returns the HTML of the message will do so using the new parser). The testing endpoint mentioned above was for testing purposes and will be discontinued.

    gharmel: feedback from the beta testing period has been incorporated - where possible - in the latest version of the parser that goes live tomorrow (Mar 8). Of course, we are constantly working on improving our HTML parser, so this work will continue based on any additional feedback that we will receive.

    Simon: I'm going to ask one of my colleagues to respond to your question as I am not sure.

  • Avatar
    Guille Padilla

    Hi Simon, the ispreview=true query ignores the CSS inline that is applied for some clients (Outlook.com, Office365 and, initially, Gmail).
    Unfortunately GMX.de, and probably the other services, modify the code before displaying it (we need still to confirm this, but that explains why use the fallback widths).
    To avoid the messages to look broken, we applied a workaround to use the mobile version in these clients (this modification is not yet available in the test endpoint).

  • Avatar
    Alan Bradford

    The newest in codeset is much appreciated, but want to note, as Pardot is our sending platform, this change to the div-based .josn methodology does break a formerly critical piece of our particular puzzle, the editable zones based on table structure in the Pardot editor. There were some very confusing moments yesterday when code seemed to drastically change once imported into Pardot.

    If an option for the legacy parser would be available (knowing the realities of maintaining both is not ideal) or an export option for .json vs table-structure would be helpful.

    Edited by Alan Bradford
  • Avatar
    Johnpurtle

    Hello!

    Does BeeFree.io have any plans to provide width adjustments inside the editor for columns? Before this HTML parser update, my coworker and I were able to go into the code and update the widths on certain columns as needed for our layouts. Now it's causing a fair bit of problems on our end to make layouts work. We're trying to avoid adding large amounts of padding inside the columns so the mobile clients won't squeeze bodies of text into a super thin column.

  • Avatar
    Massimo Arrigoni

    To answer two of the comments:

    * Alan: unfortunately there are no plans to keep supporting the old HTML parser, at least right now. We also do not plan - at least at this point in time - to allow you to pick between generating HTML that is table-based (old parser) vs. DIV-based (new parser). Sorry that I don't have better news.

    * John: column widths are tricky because of the need for the layout to work on a mobile device, which typically means stacking content. Stacking & editable column widths do not work well together. We don't have plans to allow you to control column width, other than via the available 1- to 4-column structures.

  • Avatar
    Dmason

    This appears to be causing a pretty large problem when importing Bee Pro html into Marketo. Our exported HTML worked fine until this change was made, now we're having a lot of issues. Unfortunately, due to this we're going to have to find another resource for our email builds.

    Edited by Dmason
  • Avatar
    Massimo Arrigoni

    With regard to the issues in Marketo, we will open a ticket and follow up with you so that we can further assist you.

  • Avatar
    Estevão Campos

    Hi, can you add the possibility to use the old parser? My Email Client (IBM Verse) cannot read the code generated by the new parser. Only if its in table

  • Avatar
    Massimo Arrigoni

    Hi Estevao, thanks for your feedback. At this point in time we are fully committed to the new HTML parser, which generates much thinner, better HTML, and resolves a number of UX problems, such as buttons that are fully clickable, instead of just the text within the button.
    That said, if we discover additional compatibility issues, we may reconsider the decision and may decide to provide support for both options.

  • Avatar
    Chad Rogez Design

    I am a Pro subscriber. I have a newsletter that I have frameworked in BEE Pro. From there we take it into constant contact. It was suppose to go out to our subscriber base of 19k two days ago. The new changes do not allow compatibility anymore and I have been put into a very bad spot. I need some direction or else availabilty to export in the old HTML parser formatting. PLEASE HELP!!!

  • Avatar
    Jscharf

    Same situation here. HTML code from Bee editor does not transfer properly into MaxBull Mailer. There appears to be a compatibility issue. I have a time sensitive blast that was supposed to go out on Monday. Client is totally pissed off. I am put in an embarrassing position that leaves me looking totally unprofessional. Need a firm answer when this will be fixed so I can try to salvage. I am scrambling for solutions. Why wouldn't you leave the previous version up and working until the new method is fully tested and bug free? Please urgently help!!

  • Avatar
    trushing

    We are a plug in user and was loading the html code into MaxBulk Mailer. The HTML parser is NOT WORKING! Please revert back to working version.

  • Avatar
    Sergio M.

    Hi Jscharf and trushing users, thank you for reporting a potential issue with the BEE HTML code in MaxBulk Mailer. Can you please contact MaxBulk Mailer support and ask them to check the BEE message on their side? The issue could be a number of reasons, as each ESP handles HTML emails differently.

    At the same time, our dev team will investigate into this. We apologize for this inconvenience again.

  • Avatar
    Jscharf

    Sergio - I am able to copy and paste HTML code from other newsletter sources into MaxBulk Mailer without any problems. Something is wrong on BEE's side. Everything was working fine until BEE launched their new HTML parser. Marketo, IBM, Constant Contact, MaxBulk Mailer etc are all experiencing the same problems. Apologizes not needed ... just need to have the problem fixed without further delay. I am losing clients as a result of this problem. Not a good situation.