Search upgraded (yet again)

Written By: - Date published: 12:47 am, March 15th, 2010 - 4 comments
Categories: admin - Tags:

My absence this weekend from moderation and posts has partially been because of going to the excellent Fabian seminar today. However it has largely been due to trying to get a workable search system running.

One that didn’t bring the whole server crashing around my ears every few months and cut the issue of everyone having to stop receiving data for up to a minute when someone used search. One that allowed comments to be accessed as easily as posts, which Search Unleashed achieved (which eventually failed because the re-index took too long). It looks like it is finally working correctly, but I’d like feedback on any errors that show up.

For the technically minded, this is running on the Sphinx search engine with the WordPress Sphinx Search Plugin, and some custom coding in the search sidebar and search page template for our new theme. It runs a delta update for changed items every 5 minutes, and a complete rebuild at 0300.

This is the third time I’ve attempted to get this system running. Each of the previous attempts have been stymied by sphinx disliking our constrained memory during peak hours. But we’ve bumped the RAM up, and it also turns out that there were some conflicting permission issues (grrr) when running the cron jobs with an active website.

Normally I’d have consigned something as annoying to configure as this to the dustbin, but the engine is blindingly fast and perfect for our scaling needs with nearly 5000 posts and 150,000 comments currently onsite. This is expected to keep rising at rates similar or higher than have been the case so far.

Sphinx can reindex our entire database in about 30 seconds (whereas Search Unleashed required hours), and will do the delta every 5 minutes in well under a second. Almost all of the delay in presenting the search result is from assembling the display rather than the search. It is a seriously good engine and has been (so far) worth the effort.

The user interface is a little primitive at present and I also need to store and get your user selected advanced settings in your cookies. But it is good enough to look for loading problems over this week. It has a number of options that will be useful for advanced searching, especially for authors like r0b…

Sphinx extended queries syntax allows the following special operators to be used:

  • AND: hello&world hello world
  • OR: hello | world
  • NOT: hello -world or hello !world
  • field search operator: @title hello @body world
  • phrase search operator: “hello world”
  • proximity search operator: “hello world”~10

Here’s an example query which uses all these operators:
“hello world” @title “example program”~5 @body python -(php|perl)

More information about extended syntax you can find at Sphinx.

The following field operators are available:

  • @title – search in post or page title
  • @author – search in post, page, comment author (still a bit flakey)
  • @body – search in post, page, comment body
  • @category – search search in blog categories

For instance, I tested such things as lprent&fabian lprent fabian, pigs-flying, and (supershitty|super-city|super city|supercity)& franklin.

Using the author you can search for f*ck @author robinsod (that allows you to exclude the people swearing at him).

If you use the advanced and turn pages and comments off, you can resolve a long-standing argument.

  • @title “helen clark” and @title “john key” will respectively tell you how often each was mentioned in the title of a post.
  • @body “helen clark” and @body “john key” will respectively tell you how often each was mentioned in the body of a post.

If I see that argument again…..

The time to run a search seems to be roughly the same regardless of how complex I make the query.

Anyway, let me know if you find any issues using the form in contact. I’ll be keeping an eye on it for the next 3 or 4 days for performance issues.

Hopefully I’ll now have more time  to write posts and tackle the minor teething issues like the misbehaving Contribute Past captcha, and less time trying to maintain a fragile search mechanism that was slowly fading under increased loads.

BTW: For those of you who were at the Fabian seminar today. I was paying attention to the panel and discussion. But I was also coding during the seminar. This had been bugging me most of the weekend to the point where I had to go to the seminar, tether the iPhone to the net, and finish the damn search page interface.

Update: Been through the site settings for the first time since it last had a server move. You’ll see a significant improvement in performance in the search, and probably in just reading the site.

Update: Found a couple of errors in the post, which you will see corrected with the original in strikeout. There are a number of enhancements to the system (like @author). I added some additional examples while testing.

But I was paying attention to the seminar as well

4 comments on “Search upgraded (yet again)”

  1. lprent 1

    Ummm, one thing it misses is the comment author name…. and post author name..

    Ok that is a problem in the query, which I’ll adjust in the sphinx.conf tomorrow sometime

    • mummybot 1.1

      Good on ya mate! 😉

      • lprent 1.1.1

        Did a burst of database optimization tonight as well. I’m pretty sure that the site will be significantly faster for reading and probably the same or slightly better for writing.

        Sphinx was loading down the system (as well as some of the other queries) so I tuned up a couple of tables with indexes based on what was showing in the slow query log. Then I tuned mysql to use more of the memory/disk/cpu tradeoff. Then did some tweaks to the php settings based on its logs.

        I’d better give up some of these late night sessions.

      • lprent 1.1.2

        Just while I remember, the HTML ins and del need some css coloring and decoration. del should be coloured, and the insertions should be a different colour.

Recent Comments

Recent Posts

  • CTU speech – DPM
    Ladies and gentlemen, NZCTU President Richard Wagstaff, members of respective unions – thank you for the invitation to speak to you today. This might be preaching to the choir, but the importance of trade unions in New Zealand’s historical arch is difficult to understate. And it is my belief that ...
    BeehiveBy beehive.govt.nz
    6 hours ago
  • Police Association Annual Conference
    "Let’s start by acknowledging that it has been a huge year. " Police Association Annual Conference James Cook Grand Chancellor Hotel Wellington Nau mai, haere mai. Tēnā koutou, tēnā koutou, ka nui te mihi, ki a koutou katoa. President of the Police Association, Chris Cahill; Members of the Association and ...
    BeehiveBy beehive.govt.nz
    8 hours ago
  • New Zealand announces a further P-3 deployment in support of UN sanctions
    Minister of Foreign Affairs Winston Peters and Minister of Defence Ron Mark have announced the New Zealand Government’s decision to again deploy a Royal New Zealand Air Force P-3K2 (P-3) maritime patrol aircraft to support the implementation of United Nations Security Council (UNSC) resolutions imposing sanctions against North Korea. New ...
    BeehiveBy beehive.govt.nz
    10 hours ago
  • New Zealand deeply concerned at developments in north-east Syria
    Foreign Affairs Minister Winston Peters says New Zealand continues to have serious concerns for peace and stability in north-east Syria. “Recent reports that hundreds of ISIS-affiliated families have fled from a camp are deeply concerning from a humanitarian and security perspective”, Mr Peters says. “While we acknowledge Turkey’s domestic security ...
    BeehiveBy beehive.govt.nz
    11 hours ago
  • Government on high alert for stink bugs
    Biosecurity Minister Damien O’Connor is warning travelling Kiwis to be vigilant as the high-season for the crop-eating brown marmorated stink bug (BMSB) is under way. “We’re on high alert to stop BMSB arriving in NZ. The high season runs until April 30 and we’ve strengthened our measures to stop stink ...
    BeehiveBy beehive.govt.nz
    12 hours ago
  • Better protections for students in halls of residence
    The Government is moving swiftly to change the law to improve the welfare and pastoral care of students living in university halls of residence and other tertiary hostels. Cabinet has agreed to several changes, including creating a new mandatory Code of Practice that sets out the duty of pastoral care ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • New trapping guide for community and expert trappers alike
    The Minister for Conservation Eugenie Sage has launched a new comprehensive trapping guide for community trappers to help them protect our native birds, plants and other wildlife, at Zealandia in Wellington today. ‘A practical guide to trapping’, has been developed by the Department of Conservation (DOC), and was launched during ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Widening Access to Contraceptives Welcomed
    Associate Health Minister Julie Anne Genter welcomes PHARMAC’s move to improve access to long-acting reversible contraception (LARCs). PHARMAC has today announced it will fund the full cost of Mirena and Jaydess for anyone seeking long term contraception, lifting previous restrictions on access to Mirena. “I welcome women having greater choices ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Major upgrade for Taranaki Base Hospital
    The Government has approved the next stage of a major redevelopment of Taranaki Base Hospital, which will deliver new and improved facilities for patients. Health Minister Dr David Clark has announced details of a $300 million dollar project to build a new East Wing at the New Plymouth hospital. It ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Extra support for rural families
    Extra funding will allow Rural Support Trusts to help farming families, says Minister for Rural Communities and Agriculture Damien O’Connor. “I know that rural families are worried about some of the challenges facing them, including the ongoing uncertainty created by the Mycoplasma bovis outbreak. “Those concerns sit alongside ongoing worries ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Howard Leaque Beekeeper programme graduation
    Thank you for the opportunity to be here to present certificates to the 16 graduates who have completed a beekeeping course delivered by the Howard League.  Let us start by acknowledging Auckland Prison’s Deputy Prison Director Tom Sherlock, and Acting Assistant Regional Commissioner of Corrections Northern Region Scott Walker - ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Finance Minister to attend APEC meetings
    Finance Minister Grant Robertson leaves this weekend to attend the APEC Finance Ministers meeting in Santiago, Chile. Discussions between APEC Finance Ministers at the meeting will include the effects of the current global economic uncertainty, risks for APEC economies and sustainable development of the region. While at APEC Grant Robertson ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Pacific languages are a source of strength, they ground us and build confidence
    The Minister for Pacific Peoples Aupito William Sio says for Pacific people, language can be a source of strength. It can help ground us and give us confidence. When we speak them, our languages provide us with an immediate and intimate access to our identity and our story - and ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Major boost to support disabled people in sport and recreation
    The Coalition Government has announced an action plan to improve the wellbeing of disabled New Zealanders by addressing inequalities in play, active recreation and sport. The initiative includes training to develop a workforce that understands the needs of children and young people with a range of impairments, advocacy for fit ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • More prefab homes to be built as red tape cut
    The construction sector is being freed up to allow more homes to be built more quickly as the Government cuts through some of the red tape of the Building Act.  “Every New Zealander deserves a warm, dry, safe home and old inefficiencies in the Building Act make building slow and ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Further details of Prince of Wales and Duchess of Cornwall visit to New Zealand
    Prime Minister Jacinda Ardern has welcomed further details on the Prince of Wales and Duchess of Cornwall’s visit to New Zealand next month. Their Royal Highnesses will visit New Zealand from 17-23 November – their third joint visit to New Zealand and first in four years. They arrive in Auckland ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • O’Connor in Thailand to push for RCEP deal
    Minister of State for Trade and Export Growth and Minister of Agriculture, Damien O’Connor, heads to Thailand today to attend the final Regional Comprehensive Economic Partnership (RCEP) Ministerial meeting, as negotiations enter their final stages. “The RCEP Agreement would anchor New Zealand in a regional agreement that covers 16 countries, ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Young Pacific people can access earning and learning opportunities in Hawke’s Bay, Otago and South...
    Pacific young people living in the Hawke’s Bay, Southland and Otago regions will have access to support services that have proved successful in helping young people find new earning and learning opportunities. “Tupu Aotearoa is about changing Pacific young peoples’ lives. Our young people are talented, they are smart, they ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Protecting wellbeing – ACC HQSC Trauma Forum
    Introduction As the Minister for ACC I thank you all for the work that you do supporting New Zealanders in their literally most vulnerable moments. From those who hold people’s lives in their hands, to the people who research technique, technology and trends, your work is highly valued. A special ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • NZ economy in good shape – notes prepared for speeches in Christchurch
    Notes prepared for speeches in Christchurch – Wednesday 9 October 2019 Today’s topic, “trends and opportunities for the New Zealand economy,” is certainly one getting a great deal of commentary at the moment. Looking across the media landscape lately you’ll notice we aren’t the only ones having this discussion. There ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • World Mental Health Day a reminder of the importance of mental health work
    Minister of Health Dr David Clark and Associate Minister of Health Peeni Henare say this year’s World Mental Health Day theme is a reminder of why the Government’s work on mental health is so important. “This year the World Federation for Mental Health has made suicide prevention the main theme ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Cultural Ministers Meeting
    Associate Arts, Culture and Heritage Minister Carmel Sepuloni will represent the government at Australia’s Meeting of Cultural Ministers in Adelaide this week. “This year’s meeting is special because New Zealand is expected to become an International Member of the Meeting of Cultural Ministers at this Australian forum,” Carmel Sepuloni said. “The meeting is an opportunity to ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • 608 claims resolved by GCCRS in first year
    The Greater Christchurch Claims Resolution Service has resolved 608 insurance and EQC claims in its first year in operation, Minister Megan Woods has announced. The government service, which celebrates its first birthday today, provides a one stop shop to help Cantabrians still battling to get their homes repaired or rebuilt ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • NZ economy in good shape
    Today’s topic, “trends and opportunities for the New Zealand economy,” is certainly one getting a great deal of commentary at the moment. Looking across the media landscape lately you’ll notice we aren’t the only ones having this discussion. There has been an increasing amount of attention paid to the outlook ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • NZTA to refocus on safety following review
    The Government is acting swiftly to strengthen NZTA’s regulatory role following a review into the Transport Agency, and Ministry of Transport’s performance as its monitor, Transport Minister Phil Twyford said today. An independent review by Martin Jenkins has found NZTA failed to properly regulate the transport sector under the previous ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Joint Cooperation Statement on Climate Change between the Netherlands and New Zealand
    The Netherlands and New Zealand have a long-standing and close relationship based on many shared interests and values. We value the rule of law, our democracies, and multilateralism.  And we value our environment – at home and globally. Right now there are major global challenges in all of these areas – ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government putting right Holidays Act underpayment in Health
    The Government is putting right a decade’s worth of underpayment to nurses, doctors and other health workers, says Health Minister Dr David Clark.  Initial sampling of District Health Boards payroll records has found that around $550-$650 million is owed to DHB staff to comply with the Holidays Act. It’s expected ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government accounts show strong economy
    A strong surplus and low debt show the economy is performing well, and means the Government is in a good position to meet the challenges of global economic uncertainty. “The surplus and low levels of debt show the economy is in good shape. This allows the Government to spend more ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Ministers approve application to expand Waihi mine
    New applications from mining company OceanaGold to purchase land in Waihi for new tailings ponds associated with its gold mines have been approved. Minister of Finance Grant Robertson and Associate Minister of Finance David Parker considered the applications under the Overseas Investment Act. Earlier this year, applications from OceanaGold to ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Tuia 250 Voyage flotilla launches with tribute to tangata whenua
    New Zealanders in Tūranganui-a-Kiwa / Poverty Bay will witness Māori, Pākehā and Pacific voyaging traditions come together today as the Tuia 250 Voyage flotilla assembles for the first time, Māori Crown Relations: Te Arawhiti Minister Kelvin Davis says. “Tuia 250 is a national commemoration and an opportunity for honest conversations ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Visit to advance trade agenda with Europe and the Commonwealth
    Minister for Trade and Export Growth David Parker leaves tomorrow for Dubai, London and Berlin for a series of meetings to advance New Zealand’s trade interests.  In Dubai he will visit New Zealand’s Pavilion at Expo 2020 where construction is underway.  There he will meet Minister of State for International Cooperation, Her ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • More cancer drugs confirmed – even more on horizon
    Confirmation that PHARMAC will fund two new cancer drugs is further evidence of the good progress the Government is making to improve the treatment of New Zealand’s leading cause of death, Health Minister David Clark says. From 1 December PHARMAC will fund alectinib (Alecensa) for ALK positive advanced non-small cell ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Boost for women in high performance sport
    An additional $2.7 million has been announced for the Government Strategy for Women and Girls in Sport and Active Recreation on the first anniversary of the strategy’s launch. Sport and Recreation Minister Grant Robertson gave the opening address to the first Sport NZ Women + Girls Summit in Wellington today, ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Parent support to help retain skilled migrants
    As part of its work to ensure businesses can get the skilled workers they need, the Coalition Government is re-opening and re-setting the Parent Category visa programme, Immigration Minister Iain Lees-Galloway says. The move will: support skilled migrants who help fill New Zealand’s skills gaps by providing a pathway for ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Senior NZDF Officer to lead Peacekeeping Mission in the Sinai Peninsula, Egypt
    Minister of Defence Ron Mark has today announced Major General Evan Williams of the New Zealand Defence Force has been selected as the commander of a significant, longstanding peacekeeping mission in the Middle East. In December, Major General Williams takes over as Force Commander for the Multinational Force and Observers ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Nurses star as Govt rebuilds health workforces
    A record number of nurses are now working to deliver health services to New Zealanders as the Government’s increased funding and new initiatives rebuild key workforces start to show results, Health Minister Dr David Clark says. •    1458 more DHB nurses since the Government took office •    106 more midwives ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New agricultural trade envoy appointed
    Farmer and former Nuffield scholar Mel Poulton has been appointed New Zealand’s Special Agricultural Trade Envoy, Minister for Trade and Export Growth, David Parker, and Minister of Agriculture, Damien O’Connor, announced today. The position supports key Government objectives, including raising the value of New Zealand agricultural goods and services. Mel is ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Pacific and Māori voyaging heritage celebrated for Tuia 250
    New Zealand’s Pacific and Māori voyaging heritage is acknowledged and celebrated today as waka of the Tuia 250 voyage flotilla arrive in Tūranga / Gisborne. “Today we celebrate Tangata Whenua, the first people of Aotearoa, and the triumphs of the voyaging tradition that brought our ancestors here from Polynesia 1000 ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Pacific languages are a root from which prosperity will grow
    “Fijian Language Week starts on Sunday and the theme reminds us how important it is that we each have something to anchor ourselves to, something that can help us pause and feel in control in a rapidly changing world,” says Minister for Pacific Peoples Aupito William Sio. “Family, culture, faith, ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • NZ Government establishes innovative, industry-focused Airspace Integration Trials Programme
    The Government is establishing an Airspace Integration Trials Programme to support the safe testing and development of advanced unmanned aircraft and accelerate their integration into the aviation system, Research, Science and Innovation Minister Megan Woods announced today. The Government will work with leading, innovative aviation industry partners to test and ...
    BeehiveBy beehive.govt.nz
    2 weeks ago