Search upgraded (yet again)

Written By: - Date published: 12:47 am, March 15th, 2010 - 4 comments
Categories: admin - Tags:

My absence this weekend from moderation and posts has partially been because of going to the excellent Fabian seminar today. However it has largely been due to trying to get a workable search system running.

One that didn’t bring the whole server crashing around my ears every few months and cut the issue of everyone having to stop receiving data for up to a minute when someone used search. One that allowed comments to be accessed as easily as posts, which Search Unleashed achieved (which eventually failed because the re-index took too long). It looks like it is finally working correctly, but I’d like feedback on any errors that show up.

For the technically minded, this is running on the Sphinx search engine with the WordPress Sphinx Search Plugin, and some custom coding in the search sidebar and search page template for our new theme. It runs a delta update for changed items every 5 minutes, and a complete rebuild at 0300.

This is the third time I’ve attempted to get this system running. Each of the previous attempts have been stymied by sphinx disliking our constrained memory during peak hours. But we’ve bumped the RAM up, and it also turns out that there were some conflicting permission issues (grrr) when running the cron jobs with an active website.

Normally I’d have consigned something as annoying to configure as this to the dustbin, but the engine is blindingly fast and perfect for our scaling needs with nearly 5000 posts and 150,000 comments currently onsite. This is expected to keep rising at rates similar or higher than have been the case so far.

Sphinx can reindex our entire database in about 30 seconds (whereas Search Unleashed required hours), and will do the delta every 5 minutes in well under a second. Almost all of the delay in presenting the search result is from assembling the display rather than the search. It is a seriously good engine and has been (so far) worth the effort.

The user interface is a little primitive at present and I also need to store and get your user selected advanced settings in your cookies. But it is good enough to look for loading problems over this week. It has a number of options that will be useful for advanced searching, especially for authors like r0b…

Sphinx extended queries syntax allows the following special operators to be used:

  • AND: hello&world hello world
  • OR: hello | world
  • NOT: hello -world or hello !world
  • field search operator: @title hello @body world
  • phrase search operator: “hello world”
  • proximity search operator: “hello world”~10

Here’s an example query which uses all these operators:
“hello world” @title “example program”~5 @body python -(php|perl)

More information about extended syntax you can find at Sphinx.

The following field operators are available:

  • @title – search in post or page title
  • @author – search in post, page, comment author (still a bit flakey)
  • @body – search in post, page, comment body
  • @category – search search in blog categories

For instance, I tested such things as lprent&fabian lprent fabian, pigs-flying, and (supershitty|super-city|super city|supercity)& franklin.

Using the author you can search for f*ck @author robinsod (that allows you to exclude the people swearing at him).

If you use the advanced and turn pages and comments off, you can resolve a long-standing argument.

  • @title “helen clark” and @title “john key” will respectively tell you how often each was mentioned in the title of a post.
  • @body “helen clark” and @body “john key” will respectively tell you how often each was mentioned in the body of a post.

If I see that argument again…..

The time to run a search seems to be roughly the same regardless of how complex I make the query.

Anyway, let me know if you find any issues using the form in contact. I’ll be keeping an eye on it for the next 3 or 4 days for performance issues.

Hopefully I’ll now have more time  to write posts and tackle the minor teething issues like the misbehaving Contribute Past captcha, and less time trying to maintain a fragile search mechanism that was slowly fading under increased loads.

BTW: For those of you who were at the Fabian seminar today. I was paying attention to the panel and discussion. But I was also coding during the seminar. This had been bugging me most of the weekend to the point where I had to go to the seminar, tether the iPhone to the net, and finish the damn search page interface.

Update: Been through the site settings for the first time since it last had a server move. You’ll see a significant improvement in performance in the search, and probably in just reading the site.

Update: Found a couple of errors in the post, which you will see corrected with the original in strikeout. There are a number of enhancements to the system (like @author). I added some additional examples while testing.

But I was paying attention to the seminar as well

4 comments on “Search upgraded (yet again) ”

  1. lprent 1

    Ummm, one thing it misses is the comment author name…. and post author name..

    Ok that is a problem in the query, which I’ll adjust in the sphinx.conf tomorrow sometime

    • mummybot 1.1

      Good on ya mate! 😉

      • lprent 1.1.1

        Did a burst of database optimization tonight as well. I’m pretty sure that the site will be significantly faster for reading and probably the same or slightly better for writing.

        Sphinx was loading down the system (as well as some of the other queries) so I tuned up a couple of tables with indexes based on what was showing in the slow query log. Then I tuned mysql to use more of the memory/disk/cpu tradeoff. Then did some tweaks to the php settings based on its logs.

        I’d better give up some of these late night sessions.

      • lprent 1.1.2

        Just while I remember, the HTML ins and del need some css coloring and decoration. del should be coloured, and the insertions should be a different colour.

Recent Comments

Recent Posts

  • Ō-Rākau Remembrance Bill passes first reading
    The return of the historic Ō-Rākau battle site to the descendants of those who fought there moved one step closer today with the first reading of Te Pire mō Ō-Rākau, Te Pae o Maumahara / The Ō-Rākau Remembrance Bill. The Bill will entrust the 9.7-hectare battle site, five kilometres west ...
    BeehiveBy beehive.govt.nz
    13 hours ago
  • Government to boost public EV charging network
    Energy Minister Simeon Brown has announced 25 new high-speed EV charging hubs along key routes between major urban centres and outlined the Government’s plan to supercharge New Zealand’s EV infrastructure.  The hubs will each have several chargers and be capable of charging at least four – and up to 10 ...
    BeehiveBy beehive.govt.nz
    13 hours ago
  • Residential Property Managers Bill to not progress
    The coalition Government will not proceed with the previous Government’s plans to regulate residential property managers, Housing Minister Chris Bishop says. “I have written to the Chairperson of the Social Services and Community Committee to inform him that the Government does not intend to support the Residential Property Managers Bill ...
    BeehiveBy beehive.govt.nz
    14 hours ago
  • Independent review into disability support services
    The Government has announced an independent review into the disability support system funded by the Ministry of Disabled People – Whaikaha. Disability Issues Minister Louise Upston says the review will look at what can be done to strengthen the long-term sustainability of Disability Support Services to provide disabled people and ...
    BeehiveBy beehive.govt.nz
    19 hours ago
  • Justice Minister updates UN on law & order plan
    Justice Minister Paul Goldsmith has attended the Universal Periodic Review in Geneva and outlined the Government’s plan to restore law and order. “Speaking to the United Nations Human Rights Council provided us with an opportunity to present New Zealand’s human rights progress, priorities, and challenges, while responding to issues and ...
    BeehiveBy beehive.govt.nz
    20 hours ago
  • Trade Minister travels to Riyadh, OECD, and Dubai
    Trade Minister Todd McClay heads overseas today for high-level trade talks in the Gulf region, and a key OECD meeting in Paris. Mr McClay will travel to Riyadh to meet with counterparts from Saudi Arabia and the Gulf Cooperation Council (GCC). “New Zealand’s goods and services exports to the Gulf region ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Education priorities focused on lifting achievement
    Education Minister Erica Stanford has outlined six education priorities to deliver a world-leading education system that sets Kiwi kids up for future success. “I’m putting ambition, achievement and outcomes at the heart of our education system. I want every child to be inspired and engaged in their learning so they ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • NZTA App first step towards digital driver licence
    The new NZ Transport Agency (NZTA) App is a secure ‘one stop shop’ to provide the services drivers need, Transport Minister Simeon Brown and Digitising Government Minister Judith Collins say.  “The NZTA App will enable an easier way for Kiwis to pay for Vehicle Registration and Road User Charges (RUC). ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Supporting whānau out of emergency housing
    Whānau with tamariki growing up in emergency housing motels will be prioritised for social housing starting this week, says Associate Housing Minister Tama Potaka. “Giving these whānau a better opportunity to build healthy stable lives for themselves and future generations is an essential part of the Government’s goal of reducing ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Tribute to Dave O'Sullivan
    Racing Minister Winston Peters has paid tribute to an icon of the industry with the recent passing of Dave O’Sullivan (OBE). “Our sympathies are with the O’Sullivan family with the sad news of Dave O’Sullivan’s recent passing,” Mr Peters says. “His contribution to racing, initially as a jockey and then ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Speech – Eid al-Fitr
    Assalaamu alaikum, greetings to you all. Eid Mubarak, everyone! I want to extend my warmest wishes to you and everyone celebrating this joyous occasion. It is a pleasure to be here. I have enjoyed Eid celebrations at Parliament before, but this is my first time joining you as the Minister ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Government saves access to medicines
    Associate Health Minister David Seymour has announced Pharmac’s largest ever budget of $6.294 billion over four years, fixing a $1.774 billion fiscal cliff.    “Access to medicines is a crucial part of many Kiwis’ lives. We’ve committed to a budget allocation of $1.774 billion over four years so Kiwis are ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Pharmac Chair appointed
    Hon Paula Bennett has been appointed as member and chair of the Pharmac board, Associate Health Minister David Seymour announced today. "Pharmac is a critical part of New Zealand's health system and plays a significant role in ensuring that Kiwis have the best possible access to medicines,” says Mr Seymour. ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Taking action on Fetal Alcohol Spectrum Disorder
    Hundreds of New Zealand families affected by Fetal Alcohol Spectrum Disorder (FASD) will benefit from a new Government focus on prevention and treatment, says Health Minister Dr Shane Reti. “We know FASD is a leading cause of preventable intellectual and neurodevelopmental disability in New Zealand,” Dr Reti says.  “Every day, ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • New sports complex opens in Kaikohe
    Regional Development Minister Shane Jones today attended the official opening of Kaikohe’s new $14.7 million sports complex. “The completion of the Kaikohe Multi Sports Complex is a fantastic achievement for the Far North,” Mr Jones says. “This facility not only fulfils a long-held dream for local athletes, but also creates ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Diplomacy needed more than ever
    Foreign Minister Winston Peters’ engagements in Türkiye this week underlined the importance of diplomacy to meet growing global challenges.    “Returning to the Gallipoli Peninsula to represent New Zealand at Anzac commemorations was a sombre reminder of the critical importance of diplomacy for de-escalating conflicts and easing tensions,” Mr Peters ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Anzac Commemorative Address, Buttes New British Cemetery Belgium
    Ambassador Millar, Burgemeester, Vandepitte, Excellencies, military representatives, distinguished guests, ladies and gentlemen – good morning and welcome to this sacred Anzac Day dawn service.  It is an honour to be here on behalf of the Government and people of New Zealand at Buttes New British Cemetery, Polygon Wood – a deeply ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Anzac Commemorative Address – NZ National Service, Chunuk Bair
    Distinguished guests -   It is an honour to return once again to this site which, as the resting place for so many of our war-dead, has become a sacred place for generations of New Zealanders.   Our presence here and at the other special spaces of Gallipoli is made ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Anzac Commemorative Address – Dawn Service, Gallipoli, Türkiye
    Mai ia tawhiti pamamao, te moana nui a Kiwa, kua tae whakaiti mai matou, ki to koutou papa whenua. No koutou te tapuwae, no matou te tapuwae, kua honoa pumautia.   Ko nga toa kua hinga nei, o te Waipounamu, o te Ika a Maui, he okioki tahi me o ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • PM announces changes to portfolios
    Paul Goldsmith will take on responsibility for the Media and Communications portfolio, while Louise Upston will pick up the Disability Issues portfolio, Prime Minister Christopher Luxon announced today. “Our Government is relentlessly focused on getting New Zealand back on track. As issues change in prominence, I plan to adjust Ministerial ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • New catch limits for unique fishery areas
    Recreational catch limits will be reduced in areas of Fiordland and the Chatham Islands to help keep those fisheries healthy and sustainable, Oceans and Fisheries Minister Shane Jones says. The lower recreational daily catch limits for a range of finfish and shellfish species caught in the Fiordland Marine Area and ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Minister welcomes hydrogen milestone
    Energy Minister Simeon Brown has welcomed an important milestone in New Zealand’s hydrogen future, with the opening of the country’s first network of hydrogen refuelling stations in Wiri. “I want to congratulate the team at Hiringa Energy and its partners K one W one (K1W1), Mitsui & Co New Zealand ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Urgent changes to system through first RMA Amendment Bill
    The coalition Government is delivering on its commitment to improve resource management laws and give greater certainty to consent applicants, with a Bill to amend the Resource Management Act (RMA) expected to be introduced to Parliament next month. RMA Reform Minister Chris Bishop has today outlined the first RMA Amendment ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Overseas decommissioning models considered
    Overseas models for regulating the oil and gas sector, including their decommissioning regimes, are being carefully scrutinised as a potential template for New Zealand’s own sector, Resources Minister Shane Jones says. The Coalition Government is focused on rebuilding investor confidence in New Zealand’s energy sector as it looks to strengthen ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Release of North Island Severe Weather Event Inquiry
    Emergency Management and Recovery Minister Mark Mitchell has today released the Report of the Government Inquiry into the response to the North Island Severe Weather Events. “The report shows that New Zealand’s emergency management system is not fit-for-purpose and there are some significant gaps we need to address,” Mr Mitchell ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Justice Minister to attend Human Rights Council
    Justice Minister Paul Goldsmith is today travelling to Europe where he’ll update the United Nations Human Rights Council on the Government’s work to restore law and order.  “Attending the Universal Periodic Review in Geneva provides us with an opportunity to present New Zealand’s human rights progress, priorities, and challenges, while ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Patterson reopens world’s largest wool scouring facility
    Associate Agriculture Minister, Mark Patterson, formally reopened the world’s largest wool processing facility today in Awatoto, Napier, following a $50 million rebuild and refurbishment project. “The reopening of this facility will significantly lift the economic opportunities available to New Zealand’s wool sector, which already accounts for 20 per cent of ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Speech to the Southland Otago Regional Engineering Collective Summit, 18 April 2024
    Hon Andrew Bayly, Minister for Small Business and Manufacturing  At the Southland Otago Regional Engineering Collective (SOREC) Summit, 18 April, Dunedin    Ngā mihi nui, Ko Andrew Bayly aho, Ko Whanganui aho    Good Afternoon and thank you for inviting me to open your summit today.    I am delighted ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Government to introduce revised Three Strikes law
    The Government is delivering on its commitment to bring back the Three Strikes legislation, Associate Justice Minister Nicole McKee announced today. “Our Government is committed to restoring law and order and enforcing appropriate consequences on criminals. We are making it clear that repeat serious violent or sexual offending is not ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New diplomatic appointments
    Foreign Minister Winston Peters has today announced four new diplomatic appointments for New Zealand’s overseas missions.   “Our diplomats have a vital role in maintaining and protecting New Zealand’s interests around the world,” Mr Peters says.    “I am pleased to announce the appointment of these senior diplomats from the ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Humanitarian support for Ethiopia and Somalia
    New Zealand is contributing NZ$7 million to support communities affected by severe food insecurity and other urgent humanitarian needs in Ethiopia and Somalia, Foreign Minister Rt Hon Winston Peters announced today.   “Over 21 million people are in need of humanitarian assistance across Ethiopia, with a further 6.9 million people ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Arts Minister congratulates Mataaho Collective
    Minister for Arts, Culture and Heritage Paul Goldsmith is congratulating Mataaho Collective for winning the Golden Lion for best participant in the main exhibition at the Venice Biennale. "Congratulations to the Mataaho Collective for winning one of the world's most prestigious art prizes at the Venice Biennale.  “It is good ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Supporting better financial outcomes for Kiwis
    The Government is reforming financial services to improve access to home loans and other lending, and strengthen customer protections, Commerce and Consumer Affairs Minister Andrew Bayly and Housing Minister Chris Bishop announced today. “Our coalition Government is committed to rebuilding the economy and making life simpler by cutting red tape. We are ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Trade relationship with China remains strong
    “China remains a strong commercial opportunity for Kiwi exporters as Chinese businesses and consumers continue to value our high-quality safe produce,” Trade and Agriculture Minister Todd McClay says.   Mr McClay has returned to New Zealand following visits to Beijing, Harbin and Shanghai where he met ministers, governors and mayors and engaged in trade and agricultural events with the New ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • PM’s South East Asia mission does the business
    Prime Minister Christopher Luxon has completed a successful trip to Singapore, Thailand and the Philippines, deepening relationships and capitalising on opportunities. Mr Luxon was accompanied by a business delegation and says the choice of countries represents the priority the New Zealand Government places on South East Asia, and our relationships in ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • $41m to support clean energy in South East Asia
    New Zealand is demonstrating its commitment to reducing global greenhouse emissions, and supporting clean energy transition in South East Asia, through a contribution of NZ$41 million (US$25 million) in climate finance to the Asian Development Bank (ADB)-led Energy Transition Mechanism (ETM). Prime Minister Christopher Luxon and Climate Change Minister Simon Watts announced ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Minister releases Fast-track stakeholder list
    The Government is today releasing a list of organisations who received letters about the Fast-track applications process, says RMA Reform Minister Chris Bishop. “Recently Ministers and agencies have received a series of OIA requests for a list of organisations to whom I wrote with information on applying to have a ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Judicial appointments announced
    Attorney-General Judith Collins today announced the appointment of Wellington Barrister David Jonathan Boldt as a Judge of the High Court, and the Honourable Justice Matthew Palmer as a Judge of the Court of Appeal. Justice Boldt graduated with an LLB from Victoria University of Wellington in 1990, and also holds ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Education Minister heads to major teaching summit in Singapore
    Education Minister Erica Stanford will lead the New Zealand delegation at the 2024 International Summit on the Teaching Profession (ISTP) held in Singapore. The delegation includes representatives from the Post Primary Teachers’ Association (PPTA) Te Wehengarua and the New Zealand Educational Institute (NZEI) Te Riu Roa.  The summit is co-hosted ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Value of stopbank project proven during cyclone
    A stopbank upgrade project in Tairawhiti partly funded by the Government has increased flood resilience for around 7000ha of residential and horticultural land so far, Regional Development Minister Shane Jones says. Mr Jones today attended a dawn service in Gisborne to mark the end of the first stage of the ...
    BeehiveBy beehive.govt.nz
    2 weeks ago

Page generated in The Standard by Wordpress at 2024-04-30T18:11:20+00:00