FeedWordPress 2010.0528: a couple of important fixes to the previous big upgrade

FeedWordPress 2010.0531 is now available for download.

I put out this update quckly because it contains fixes to a couple of important issues that y’all helped me to discover in early user reports back about FeedWordPress 2010.0528. Most of the issues that you reported are issues that are linked, in one way or another, to memory consumption, and to running up against the memory constraints provided by PHP. I will be working on some changes over the next several days to FeedWordPress’s internal handling of feed data, and the new MagpieRSS compatibility layer, which I hope will help to alleviate the problems with memory consumption, but I wanted to make sure that this bugfix release was issued as quickly as possible in order to help avert a couple of other problems — database performance problems during updates, and problems with certain syndicated items not having the syndication meta-data properly attached. (This would result in posts appearing as if they were authored locally, without meta-data about syndication source, with permalinks pointing back to the aggregator website, etc.) Both of these issues were actually caused by a common problem, which I believe has been resolved by the changes in this release. I apologize for the difficulties, and want to thank the users who sent me the detailed, informative bug reports that helped me identify the part of the code that needed to be tested, track down the issue, and eliminate the bug. As a result of one part of this fix, it will be much more difficult in the future for any error condition to result in syndicated posts to lose their custom fields or to end up being improperly sourced in the future.

Here’s a quick breakdown of the major changes since the release of FeedWordPress 2010.0531:

  • PERMALINK / CUSTOM FIELDS PROBLEM RESOLVED: An issue in 2010.0528 caused some significant performance problems when updating feeds, and caused
    some posts to be imported without the proper syndication-related
    meta-data being attached (thus causing permalinks to point back to the
    aggregator website rather than to the source website, among other
    problems). This problem has been resolved (and a fix has been applied
    which will resolve the problem for any posts affected by this problem,
    if the original post is recent enough to still be available on the feed).

  • UI: The “Back End” section has been split into two separate sections
    “Performance” (dealing with caching, database index, and other
    performance tweaks), and “Diagnostics” (dealing with debug mode,
    update logging, and a number of new diagnostic tests which I will be
    rolling out over the next few releases).

  • Several minor interface bugs and PHP warning notices have also been eliminated.

Download and enjoy! If you have any issues with the release, or any questions I can help answer, or if there is anything that you would like to see included in a future release, please use the comments form or drop me a line to let me know about it. If you have an issue to report, please be sure to tell me what version of FeedWordPress you’re using, what version of WordPress you’re using it with, and try to tell me, as clearly as possible, what you’re trying to accomplish, what you expect to see, and what you see instead when you try to accomplish it (including any error messages you may be seeing). See also Troubleshooting for more details.

Please remember that your generous gifts to the project tip jar make ongoing development, quick fixes and timely support for FeedWordPress possible.

FeedWordPress 2010.0528: Like four updates in one!

FeedWordPress 2010.0528 is now available for download.

This is a major release with many changes, fixes, and new features. Actually, this post will be covering a lot more than just what went into the most recent public release — if you compared the version numbers in the Download box to the blog posts, you may have noticed that I put out a good three public releases that didn’t have corresponding announcements on the blog. (Versions 2009.1111, 2009.1112, and 2010.0127. The reasons, basically, had to do with major releases being postponed for a couple months while I worked on some major internal overhauls, and then ending up with some important stuff to push out, but not having time to get the package’s ChangeLog up to date. Since the announcement posts are based on the ChangeLog, the announcement posts didn’t go up either.

So, here is my big, fat list of changes since FeedWordPress 2009.0707, the last version that I wrote a blog post about. Some of the most important changes are not obviously visible to end-users, but will make a very big difference: in particular, FeedWordPress is no longer depends on MagpieRSS for feed parsing; it now uses the SimplePie module included with versions of WordPress since version 2.8. SimplePie is much more up to date than MagpieRSS, handles modern feed formats just fine, and is designed with a great deal of flexibility in mind, so the transition should eliminate some common causes of parsing and update errors; it also means that I no longer have to worry about maintaining an unauthorized upgrade to WordPress core modules, and this should eliminate one of the most common causes of sudden unexpected feed errors (the Upgrade Downgrade problem will no longer exist). SimplePie’s greater extensibility will also allow many more options in the future for developing extensions or alternatives for users who prefer different arrangements with regard to feed caching or other tasks related to the parser.

(Don’t worry, if you use existing filters and add-ons that depend on MagpieRSS’s format for feed data. This version includes a compatibility layer that will mock up Magpie’s data format from SimplePie objects, and ensure that existing add-ons and filters do not break.)

Some other important new features include syntax for storing feed meta-data in custom fields for syndicated posts; allowing you to protect manual edits to syndicated posts from being overwritten by updates from the feed; settings to help you control update scheduling; more resilient handling of dates (so that feed formats with unusual or broken date formats should no longer result in posts being dated back to 1969 or 1970), and a number of other interface tweaks, compatibility fixes, and some bug fixes around the edges. Here’s a breakdown of the major changes since the release of FeedWordPress 2009.0707.

  • SIMPLEPIE IS NOW USED TO PARSE FEEDS; NO MORE MAGPIERSS UPGRADES NEEDED:
    One of the biggest changes in this release is that FeedWordPress no
    longer depends on MagpieRSS to parse feeds, and has switched to the much
    more up-to-date and flexible SimplePie feed parser, which is included as
    a standard part of WordPress versions 2.8 and later. Using SimplePie will
    hopefully allow for better handling of feeds going further, and will
    allow me greater flexibility in determining how exactly the feed parser
    will operate. It also means that FeedWordPress no longer requires
    special upgrades to the WordPress core MagpieRSS files, and should
    eliminate quite a bit of complexity.

  • MAGPIERSS COMPATIBILITY LAYER FOR EXISTING FILTERS AND ADD-ONS: However,
    I have also implemented a compatibility layer to ensure that existing
    filters and add-ons for FeedWordPress which depended on the MagpieRSS
    data format should not be broken by the switch to SimplePie. Going
    forward, I recommend that new filters and add-ons be written to take
    advantage of the SimplePie object representations of items, feeds, etc.,
    rather than the MagpieRSS arrays, but the MagpieRSS arrays will still
    be available and older filters should continue to work as they have in
    the past.

  • COMPATIBILITY WITH WORDPRESS 2.9.x and 3.0: This release has been tested
    for the existing WordPress 2.9.x branch and with the upcoming release of
    WordPress 3.0. Changes in the user interface JavaScript between WordPress
    2.8.x and WordPress 2.9 caused the tag box interface element to break in
    the Syndication –> Categories & Tags settings page; changes in the API
    functions for adding new authors caused fatal errors under certain
    conditions in WordPress 3.0. These breakages have been fixed.

  • DROPPED LEGACY SUPPORT FOR WORDPRESS PRIOR TO 2.8: Because SimplePie is
    not included with versions of WordPress prior to 2.8, I have chosen to
    drop legacy support for WordPress versions 1.5 through 2.7. If you are
    using FeedWordPress with a version of WordPress before 2.8, you will
    have to upgrade your installation of WordPress in order to take
    advantage of this release.

  • PHP 5.3 COMPATIBILITY: A couple of compatibility issues, which were
    causing fatal errors amd ugly warnings for users of PHP 5.3,
    have been eliminated.

Features and Processing

  • INTERFACE REORGANIZATION: The interface restructuring, began with
    Version 2009.0612, has been completed. Catch-all settings pages have
    been eliminated entirely for pages that cover each aspect of handling
    a feed: Feeds & Updates, Posts & Links, Authors, Categories & Tags,
    and Back End handling of the database and diagnostic information.
    Extensive new interface hooks allow add-on modules to significantly
    change or extend the FeedWordPress admin interface and workflow.

  • STORING INFORMATION FROM THE FEED IN CUSTOM FIELDS: Many users
    have written to request the ability to store information from elements
    in the feed in a custom field on each post. (So that, for example, if
    post includes a itunes:duration element, you could store the contents
    in a Custom Field called duration on the post (for a Theme to access
    later). The Custom Post Settings under Syndication –> Posts & Links now
    allow you to access any item or feed tag, using a syntax similar to
    a much-simplified version of XPath. See Posts & Links settings for
    details.

  • UPDATE-FREEZING ON MANUALLY EDITED POSTS: FeedWordPress now allows you
    to mark posts that have been manually edited, so that the changes you
    make will not be overwritten by later updates from the feed. If you make
    manual edits to a particular post, just check the “Manual editing”
    checkbox in order to protect your changes from being overwritten. If you
    want to block all posts from being updated after they are imported
    for the first time, a new “Updated Posts” setting in Posts & Links
    allows you to freeze all posts from a particular feed, or all syndicated
    posts.

  • SETTING: FEED-BY-FEED SETTINGS FOR WHERE PERMALINKS POINT TO: You’ve
    always been able to tell FeedWordPress whether permalinks for posts
    should point to the original source of the story or the local copy. Now
    you can choose different policies for different feeds, instead of one
    global policy for all feeds. (Of course, you can still use a global
    default if you prefer.)

  • SETTING: USER CONTROL OVER TIMING BASIS. You can now determine the
    schedule on which feeds are considered ready to poll for updates —
    by default feeds become ready for polling after about 1 hour. You can
    now increase or decrease the time window under Syndication –> Feeds &
    Updates. (However, please pay CAREFUL ATTENTION to the recommendations
    and DO NOT set the scheduling lower than 60 minutes unless you are
    ABSOLUTELY SURE that you have specific permission from webmaster who
    provides that specific feed to poll more frequently than that. If you
    set this too low (and about 60 minutes is the polite minimum if you
    haven’t been given a different figure), most webmasters will consider
    the frequent hits on their server as rude, or even downright abusive.

  • OTHER SETTINGS: New settings also include the ability to stop FWP from
    resolving relative URLs within syndicated content, and the ability to
    choose whether FeedWordPress should indicate the comment feed from the
    original source, or the local comment feed, when providing the comment
    feed URL for a syndicated post.

Parsing

  • BETTER DATE HANDLING — FEWER FLASHBACKS TO 1969 and 1970: FeedWordPress
    has made some bugfixes and some improvements in the logic for parsing
    dates. This should allow FeedWordPress to correctly parse more dates in
    more feeds; and, in the last resort, when FeedWordPress fails to
    correctly parse a date, to fall back to a more intelligent default. This
    should hopefully avoid most or all error conditions that have resulted
    in articles being erroneously dated to the dawn of the Unix epoch
    (31 December 1969 or 1 January 1970).

  • FULL-TEXT “EXCERPTS” NOW PROPERLY SHORTENED. Based on a straightforward
    reading of the existing RSS specs, it’s reasonable for the
    rss:description element to be read as a plaintext summary or excerpt for
    the item containing the description — with the full text of the item,
    if available, in another, better-suited element, such as the de facto
    standard content:encoded extension element. The problem is that uses of
    RSS rarely have much to do with anything like a straightforward reading
    of the specs. As a result, many actual RSS producers in the wild put the
    full text of the article in a description element. But since
    FeedWordPress has treated this text as a summary, this produces
    aggregated posts with lengthy “excerpts” containing the full text of the
    article. This release of FeedWordPress fixes the problem by doing a
    little digging before treating rss:description as a summary: if the
    description element is used properly as a plain text summary, then
    FeedWordPress will take the summary provided by the feed, rather than
    recreating its own excerpt from the full text; but if an RSS item has no
    full-text element other than description, FeedWordPress will treat the
    description element as the full text of the article, and generate a
    shortened excerpt automatically from that text.

API

  • TEMPLATE API: new template tags get_local_permalink() and
    the_local_permalink() allow you to access the permalink for a post on
    your aggregator site, even when FeedWordPress is rewriting permalinks to
    point to the original source site.

  • NEW HOOKS FOR ADD-ONS AND FILTERS: I have added a number of new hooks
    which allow add-on modules to filter more precisely, gather information
    at more points, and to enhance the FeedWordPress admin interface. For
    a list of new hooks and documentation, see the FeedWordPress
    documentation wiki at
    http://feedwordpress.radgeek.com/wiki/add-ons-and-filters

  • FILTER API: A number of new utility methods have been added to the
    SyndicatedPost class to make it easier for filters and add-ons to

  • FILTER API: Globals $fwpchannel and $fwpfeedmeta DEPRECATED. These
    global variables, originally introduced to allow filters access to
    information about the source feed in syndicated_item filters (which
    were passed in through global variables rather than as parameters
    because of a bug in WP 1.5 which was then fixed in 1.5.1) have been
    DEPRECATED. If you have any filters or add-ons which still depend on
    these global variables, you should see about fixing them to access data
    about the source feed using the SyndicatedPost::link element instead.
    For documentation, see the FeedWordPress documentation wiki at
    http://feedwordpress.radgeek.com/wiki/syndicatedpost and
    http://feedwordpress.radgeek.com/wiki/syndicatedlink.

  • DIAGNOSTICS: I’ve included a number of new diagnostic options and
    messages, which should allow an experienced user to better investigate
    any problems that may crop up.

Bug Fixes

  • BUGFIX: & IN PERMALINKS NO LONGER CAUSING ATOM OR HTML VALIDATION
    ERRORS:
    Many users reported an issue in which syndicating a feed with
    special XML characters in the URLs (& was the most common, since it is
    used to separate HTTP GET parameters) would cause the aggregator’s
    feeds to produce invalid (malformed) XML. This update addresses the
    issue in Atom feeds. Unfortunately, it has not been technically possible
    to address the problem in RSS 2.0 feeds, due to limitations on
    WordPress’s internal templates for RSS feeds.

  • BUGFIX: BROKEN URLS IN “POPULAR POSTS” AND SIMILAR PLUGINS SHOULD NO
    LONGER BE BROKEN.
    A number of users noticed an issue where plugins and
    templates that listed posts in locations outside of the post loop
    (for example, “Popular Posts”-style plugins that listed posts in the
    sidebar), often produced the wrong URL for post links. (Typically, all
    the posts listed would get the same wrong URL.) This should now be
    fixed. Thanks to Björn for sending in a quick fix!

  • MINOR BUGFIXES: This release includes a number of fixes to minor bugs
    and compatibility issues, including: silent failures of the “Syndicate”
    button, “Illegal Offset Type” error messages from MagpieRSS, and others.

Enjoy! As always, if you have any issues with the release, or any questions I can help answer, or if there is anything that you would like to see included in a future release, please use the comments form or drop me a line to let me know about it.

I want to thank everyone who has contributed to the project tip jar over the past several months. Your generous gifts made it possible for me to take a lot of time out from freelance work in order to push through some really major changes to FeedWordPress over the past few months, and your ongoing support, besides being very flattering, goes a long way to help make development, support, and timely fixes possible. Thank you all!