FeedWordPress 0.97: one small increment for the revision number, one giant leap for the software

Update 2007-11-21: FeedWordPress 0.97 is now out of date. You can download the latest release — 0.991 at the time of this writing — from the project homepage.

Huzzah! FeedWordPress 0.97 is now available for download (also in zip format, if you’re in to that kind of thing). Enjoy, and please feel free to report any problems you have with the software or the documentation to me in the comments section below or by e-mail.

Version 0.97 represents major changes, I hope for the better, on several fronts in FeedWordPress. (If I weren’t running out of revision numbers between the current version and 1.0, I would have added a lot more than 0.01 to the version. Stupid decimal notation.) This release will address several long-standing bugs, and hopefully also make other bugs easier to diagnose and to fix. First, the minor inconvenience that this update will impose on you if you are upgrading:

  • INSTALLATION PROCEDURE: Some of the changes between 0.96 and 0.97
    require upgrades to the meta-data stored by FeedWordPress to work
    properly. Thus, if you are upgrading from 0.96 or earlier to 0.97, most
    FeedWordPress operations (including updates and template functions)
    WILL BE DISABLED until you run the upgrade procedure. Fortunately,
    running the upgrade procedure is easy: just go to either Options –>
    Syndication or Links –> Syndicated in the WordPress Dashboard and press
    the button.

Now, the fun stuff. The most important are the new, humane interface for setting feed-specific settings (check it out under Links –> Syndicated –> Edit … no more nerding around with key-value pairs!), Atom 1.0 support, and the overhaul of update-feeds.php. Details follow:

  • FEED FORMAT SUPPORT: Support has been added for the Atom 1.0 IETF
    standard. Several other elements are also newly supported
    (dcterms:created, dcterms:issued, dcterms:modified, dc:identifier,
    proper support for the RSS 2.0 guid element, the RSS 2.0 author element,
    the use of Atom author or Dublin Core dc:creator constructs at the feed
    level to identify the author of individual items, etc.)

    N.B.: full support of several Atom 1.0 features, such as categories
    and enclosures, requires you to install the optional rss-functions.php
    upgrade in your wp-includes directory.

  • BUG FIX: Running update-feeds.php from command line or crontab
    returned “I don’t syndicate…” errors. It turns out that WordPress
    sometimes tramples on the internal PHP superglobals that I depended on
    to determine whether or not the script was being invoked from the
    command line. This has been fixed (the variables are now checked
    before WordPress can trample them). Note that update-feeds.php has
    been thoroughly overhauled anyway; see below for details.

  • BUG FIX: Duplicate categories or author names. Fixed two bugs that could
    create duplicate author and/or category names when the name contained
    either (a) certain international characters (causing a mismatch between
    MySQL and PHP’s handling of lowercasing text), or (b) characters that
    have a special meaning in regular expressions (causing MySQL errors when
    looking for the author or category due to regexp syntax errors). These
    should now be fixed thanks to careful escaping of names that go into
    regular expressions and careful matching of lowercasing functions
    (comparing results from PHP only to other results from PHP, and results
    from MySQL only to other results from MySQL).

  • BUG FIX: Items dated Decembr 31, 1969 should appear less often. The
    function for parsing W3C date-time format dates that ships with
    MagpieRSS can only correctly parse fully-specified dates with a
    fully-specified time, but valid W3C date-time format dates may omit the
    time, the day of the month, or even the month. Some feeds in the wild
    date their items with coarse-grained dates, so the optional
    rss-functions.php upgrade now includes a more flexible parse_w3cdtf()
    function that will work with both coarse-grained and fully-specified
    dates. (If parts of the date or the time are omitted, they are filled in
    with values based on the current time, so ‘2005-09-10’ will be dated to
    the current time on that day; ‘2004’ will be dated to this day and time
    one year ago.

    N.B.: This fix is only available in the optional rss-functions.php
    upgrade.

  • BUG FIX: Evil use of HTTP GET has been undone. The WordPress interface
    is riddled with inappropriate (non-idempotent) uses of HTTP GET queries
    (ordinary links that make the server do something with significant
    side-effects, such as deleting a post or a link from the database).
    FeedWordPress did some of this too, especially in places where it aped
    the WordPress interface (e.g. the “Delete” links in Links –>
    Syndicated). That’s bad business, though. I’ve changed the interface so
    that all the examples of improper side-effects that I can find now
    require an HTTP POST to take effect. I think I got pretty much
    everything; if there’s anything that I missed, let me know.

    Further reading: Ruby 2005-05-06: This Stuff Matters (http://www.intertwingly.net/blog/2005/05/06/This-Stuff-Matters)

  • BUG FIX: Categories applied by cats setting should no longer prevent
    category-based filtering from working. In FeedWordPress, you can (1)
    apply certain categories to all syndicated posts, or all posts from
    a particular feed; and (2) filter out all posts that don’t match one
    of the categories that are already in the WordPress database (allowing
    for simple category-based filtering; just load up WordPress with the
    categories you want to accept, and then tell FeedWordPress not to create
    new ones). However, the way that (1) and (2) were implemented meant that
    you couldn’t effectively use them together; once you applied a known
    category to all syndicated posts from a particular feed, it meant that
    they’d have at least one familiar category (the category or categories
    you were applying), and that would get all posts past the filter no
    matter what categories they were originally from.

    Well, no longer. You can still apply categories to all syndicated posts
    (using either Syndication –> Options, or the feed-level settings under
    Links –> Syndicated). But these categories are not applied to the post
    until after it has already passed by the “familiar categories” filter.
    So now, if you want, you can do category filtering and then apply as
    many categories as you please to all and only posts that pass the filter.

  • BUG FIX: Other minor typos and HTML gaffes were fixed along the way.

  • PERFORMANCE: getfeedmeta() no longer hits the database for information
    on every call; it now caches link data in memory, so FeedWordPress only
    goes to the database once for each syndicated link. This may
    substantially improve performance if your database server resources
    are tight and your templates make a lot of use of custom settings from
    getfeedmeta().

  • API CHANGE: Link ID numbers, rather than RSS URIs, are now used to
    identify the feed from which a post is syndicated when you use template
    functions such as getfeedmeta(). The practical upshot of this is you
    can switch feeds, or change the feed address for a particular syndicated
    site, without breaking your templates for all the posts that were
    syndicated from the earlier URI.

  • API CHANGE: if you have plugins or templates that make use of the
    getfeedmeta() function or the $fwpfeedmeta global, note that the
    data formerly located under the uri and name fields is now located
    under the link/uri field and the link/name field, respectively. Note
    also that you can access the link ID number for any given feed under the
    global $fwp
    feedmeta[1] (in plugins) or
    getfeedmeta(‘link/id’) (in a template in post contexts).

  • FEATURE: the settings for individual feeds can now be edited using a
    humane interface (where formerly you had to tweak key-value pairs in the
    Link Notes section). To edit settings for a feed, pick the feed that you
    want under Links –> Syndicated and click the Edit link.

  • FEATURE: The “Unsubscribe” button (formerly “Delete”) in Links –>
    Syndicated now offers three options for unsubscribing from a feed: (1)
    turning off the subscription without deleting the feed data or affecting
    posts that were syndicated from the feed (this works by setting the Link
    for the feed as “invisible”); (2) deleting the feed data and all of the
    posts that were syndicated from the feed; or (3) deleting the feed data
    and keeping the posts that were syndicated from the feed
    setting the Link to “Invisible” (meaning that it will not be displayed
    in lists of the site links on the front page, and it won’t be checked
    for updates; (2) deleting the Link and all of the posts that were
    syndicated from its feed; or (3) deleting the feed data but keeping the
    posts that were syndicated (which will henceforward be treated as if
    they were local rather than syndicated posts). (Note that (1) is usually
    the best option for aggregator sites, unless you want to clean up the
    results of an error or a test.)

  • FEATURE / BUG FIX: If you have been receiving mysterious “I don’t
    syndicate…”, or “(local) HTTP status code was not 200”, or “(local)
    transport error – could not open socket”, or “parse error – not well
    formed” errors, then this update may solve your problems, and if it does
    not solve them, it will at least make the reasons for the problems
    easier to understand. That’s because I’ve overhauled the way that
    FeedWordPress goes about updating feeds.

    If you use the command-line PHP scripting method to run scheduled
    updates, then not much should change for you, except for fewer
    mysterious errors. If you have done updates by sending periodic HTTP
    requests to http://your-blog.com/path/wp-content/update-feeds.php,
    then the details have changed somewhat; mostly in such a way as to make
    things easier on you. See the README file or online documentation on
    Staying Current for the details.

  • FEATURE: FeedWordPress now features a more sophisticated system for
    timed updates. Instead of polling every subscribed feed for updates
    each time update-feeds.php is run, FeedWordPress now keeps track of
    the last time it polled each feed, and only polls them again after a
    certain period of time has passed. The amount of time is normally set
    randomly for each feed, in a period between 30 minutes and 2 hours (so
    as to stagger updates over time rather than polling all of the feeds at once. However, the length of time between updates can also be set
    directly by the feed, which brings us to …

  • FEATURE: FeedWordPress now respects the settings in the ttl and
    Syndication Module RSS elements. Feeds with these elements set will not
    be polled any more frequently than they indicate with these feeds unless
    the user manually forces FeedWordPress to poll the feed (see Links –>
    Syndicated –> Edit settings).

76 thoughts on “FeedWordPress 0.97: one small increment for the revision number, one giant leap for the software

  1. I could access to the shell prompt, and into the editor.

    Now for the next 2 questions, and sorry if I sound so dumb.

    The user and pass are the ones you use to get to the server or to the blog?

    If the user is “hipo” and pass “me”, the line must be:

    */30 * * * * curl –silent –user hipo:me http://xyz.com/wp-content/update-feeds.php –data update=quiet

    if that´s correct?

    Thanks

  2. hipocratico:

    Don’t worry about asking — like most Unix things, cron is very useful but also pointlessly arcane.

    The username and password that you should use are a username and password that will log you in to the WordPress Dashboard. Not the username and password you use to log in to the shell prompt. The line you have above is basically correct; but you should use two dashes in front of the silent, user, and data options. (Sorry about that! I hadn’t noticed that WordPress was converting double dashes into a single em dash until after I saw your comment. So the line should be:

    */30 * * * * curl --silent --user hipo:me http://xyz.com/wp-content/update-feeds.php --data update=quiet
    

    Hope this helps.

  3. best web hosting provider:

    > btw, it seems like the end of the curl line should read: […]

    Yep! My mistake.

    So here’s the correct line. Backslashes represent the continuation of a single line, not literal backslashes:

    /usr/bin/curl http://xyz.com/blog/wp-content/update-feeds.php \
         -d update=quiet \
         -d log=peter \
         -d pwd=abc
    

    Thanks for the typo catch.

  4. I just can´t get to update. I used “crontab -e”, the text editor appeared, copy-paste the line you wrote, with the correct user-pass. I set it to 10 (10 mins). Nothing happened.

    I save the line after I exit the editor. I think there´s no problem there.

    Do I have to CHMOD the archives?

    Is there hope for me?

  5. Excellent plugin. Had it up and working in a few minutes.

    I’d like to make a suggestion. When syndicating we have the permalink option set to our website as I want to encourage comments about the feed on our site as well as maintain the visitor longer.

    However I would also like to show the link that the RSS article originally linked to. So can this be added as an option to show within the post body the original feed link.

  6. Dear Charles,

    I must be doing something wrong. I’ve been trying to figure this out for a few days but I cannot.

    After the fresh installation of feedwordpress 0.97, I am unable to view the “Syndication” tab under option. Can you please tell me what I should look into? (I’m using wpmu.)

    Thank you alot! Calnet

  7. Hi Rad,

    Is there a way in which I can limit the amount of post that feedwordpress post at any one time?

    For example, my sports blog pulls in a lot of posts, sometimes 20 odd. I would really like to adjust the amount of posts that are posted or set a maximum limit.

    thanks as always john

  8. Hi Rad,

    I have a problem with the timestamp of the feeds. It seems like it’s odd by 6 hours. But it is actually the local servertime, so I wonder if there’s something wrong with the feeds. I am using both rss2 and blogspots atom-feed.

    Still I guess it should be possible to configure the “fallback-offset” to some other value than the local servertime, but how/where ???

  9. I am having consistent problems with the crontab and getting the job to go thru.

    With the original setting you provide,

    8 * * * * cd $HOME/http://yoursite.com/etc

    the CRON mailer sends me a message stating that the PHP command does not work.

    with the other cron jobs you suggest,

    curl –user login:pass http://mysite.com/blog/wp-content/update-feeds.php -d update=quiet

    and

    /usr/bin/curl http://xyz.com/blog/wp-content/update-feeds.php -d update=quiet -d log=peter -d log=ab (or pwd=ab, either one)

    I am getting the following from the mailer when it attempts:

    curl: option -d: requires parameter curl: try ‘curl –help’ or ‘curl –manual’ for more information

    All of the necessary CHMOD stuff is done. I have also tried using the secretword to no avail. The commands work from within the bash shell if I run them from a command line, but the cron job will not complete.

    Any suggestions?

  10. This is a great plugin! I have one problem and one request:

    However, when I run cron and it updates, it ignores my manual setting. Am I doing something wrong?

    • Speaking of cron, is there a way to to display, outside the loop, the timestamp of the cron? I would like to show at the top of my site “Updated 1/2/2005 at 9:00 EST” or words to that effect.
  11. One other thing (sorry!). I am also having the time offset problem. The timestamp on my feed posts are +6hrs, which is my server’s time (UTC). Is there a way to offset this so it matches my timezone?

  12. OK, the trick is done in feedwordpress.php , approx. line 2070 and 2071, where you add an offset. (I and the majority of the users live in Denmark, which is GMT+1 and the server is somewhere in USA)

    I replaced

    $post[‘postdate’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘issued’]); $post[‘postmodified’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘modified’]);

    with

    $post[‘postdate’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘issued’]+66060); $post[‘postmodified’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘modified’]+66060);

    Not being a php-programmer, it took me a while..

    Next version could have define-statement with SERVER_OFFSET set to 0 , and the lines:

    $post[‘postdate’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘issued’]+SERVEROFFSET); $post[‘postmodified’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘modified’]+SERVEROFFSET);

    Then it should be easy to make the adjustments.

    Hope it works..

  13. Well, it is hard to write messages in this place, I mean: 6 times 60 times 60 aka 6x60x60 with the x’s replaced with *’s

  14. Absolutely great plugin!

    I’ve been doing some hacking around and finnaly got it working with WordPress 2.0 RC1. I would like to suggest a change to the code though: Users are currently identified by their name which means if two users, from two different feeds, happen to have the same username they will be taken for the same user when aggregated into wordpress. My suggestion would be to compose the username created by FWP for each user with the author’s name and author’s url, so that they would look something like this:

    jhondoe.somedomain.com/weblog

    For a user who’s original username is “jhondoe” and who’s weblog URL is actually htp://somedomain.com/weblog

    I’m not sure if I’m missing something though, it’s quite possible this is just silly 🙂

    any toughts?

    Cheers

  15. I’m having a new problem with the plugin..it seems to be not updating all the feeds it used to..it seems to randomly not update them, even when I manually hit update-feeds.php

  16. Steve, what happens if you schedule the “Next update” for “ASAP” before hitting update-feeds.php?

  17. There’s definitely a problem..when I do the manual update, it lists three or four sites only, and I know that lots more sites in my list have udpated feeds. It’s as if update-feeds.php times out or something. If update each one manually, it works..update All is where it hangs up.

Comments are closed.