web analytics

Search upgraded (yet again)

Written By: - Date published: 12:47 am, March 15th, 2010 - 4 comments
Categories: admin - Tags:

My absence this weekend from moderation and posts has partially been because of going to the excellent Fabian seminar today. However it has largely been due to trying to get a workable search system running.

One that didn’t bring the whole server crashing around my ears every few months and cut the issue of everyone having to stop receiving data for up to a minute when someone used search. One that allowed comments to be accessed as easily as posts, which Search Unleashed achieved (which eventually failed because the re-index took too long). It looks like it is finally working correctly, but I’d like feedback on any errors that show up.

For the technically minded, this is running on the Sphinx search engine with the WordPress Sphinx Search Plugin, and some custom coding in the search sidebar and search page template for our new theme. It runs a delta update for changed items every 5 minutes, and a complete rebuild at 0300.

This is the third time I’ve attempted to get this system running. Each of the previous attempts have been stymied by sphinx disliking our constrained memory during peak hours. But we’ve bumped the RAM up, and it also turns out that there were some conflicting permission issues (grrr) when running the cron jobs with an active website.

Normally I’d have consigned something as annoying to configure as this to the dustbin, but the engine is blindingly fast and perfect for our scaling needs with nearly 5000 posts and 150,000 comments currently onsite. This is expected to keep rising at rates similar or higher than have been the case so far.

Sphinx can reindex our entire database in about 30 seconds (whereas Search Unleashed required hours), and will do the delta every 5 minutes in well under a second. Almost all of the delay in presenting the search result is from assembling the display rather than the search. It is a seriously good engine and has been (so far) worth the effort.

The user interface is a little primitive at present and I also need to store and get your user selected advanced settings in your cookies. But it is good enough to look for loading problems over this week. It has a number of options that will be useful for advanced searching, especially for authors like r0b…

Sphinx extended queries syntax allows the following special operators to be used:

  • AND: hello&world hello world
  • OR: hello | world
  • NOT: hello -world or hello !world
  • field search operator: @title hello @body world
  • phrase search operator: “hello world”
  • proximity search operator: “hello world”~10

Here’s an example query which uses all these operators:
“hello world” @title “example program”~5 @body python -(php|perl)

More information about extended syntax you can find at Sphinx.

The following field operators are available:

  • @title – search in post or page title
  • @author – search in post, page, comment author (still a bit flakey)
  • @body – search in post, page, comment body
  • @category – search search in blog categories

For instance, I tested such things as lprent&fabian lprent fabian, pigs-flying, and (supershitty|super-city|super city|supercity)& franklin.

Using the author you can search for f*ck @author robinsod (that allows you to exclude the people swearing at him).

If you use the advanced and turn pages and comments off, you can resolve a long-standing argument.

  • @title “helen clark” and @title “john key” will respectively tell you how often each was mentioned in the title of a post.
  • @body “helen clark” and @body “john key” will respectively tell you how often each was mentioned in the body of a post.

If I see that argument again…..

The time to run a search seems to be roughly the same regardless of how complex I make the query.

Anyway, let me know if you find any issues using the form in contact. I’ll be keeping an eye on it for the next 3 or 4 days for performance issues.

Hopefully I’ll now have more time  to write posts and tackle the minor teething issues like the misbehaving Contribute Past captcha, and less time trying to maintain a fragile search mechanism that was slowly fading under increased loads.

BTW: For those of you who were at the Fabian seminar today. I was paying attention to the panel and discussion. But I was also coding during the seminar. This had been bugging me most of the weekend to the point where I had to go to the seminar, tether the iPhone to the net, and finish the damn search page interface.

Update: Been through the site settings for the first time since it last had a server move. You’ll see a significant improvement in performance in the search, and probably in just reading the site.

Update: Found a couple of errors in the post, which you will see corrected with the original in strikeout. There are a number of enhancements to the system (like @author). I added some additional examples while testing.

But I was paying attention to the seminar as well

4 comments on “Search upgraded (yet again)”

  1. lprent 1

    Ummm, one thing it misses is the comment author name…. and post author name..

    Ok that is a problem in the query, which I’ll adjust in the sphinx.conf tomorrow sometime

    • mummybot 1.1

      Good on ya mate! 😉

      • lprent 1.1.1

        Did a burst of database optimization tonight as well. I’m pretty sure that the site will be significantly faster for reading and probably the same or slightly better for writing.

        Sphinx was loading down the system (as well as some of the other queries) so I tuned up a couple of tables with indexes based on what was showing in the slow query log. Then I tuned mysql to use more of the memory/disk/cpu tradeoff. Then did some tweaks to the php settings based on its logs.

        I’d better give up some of these late night sessions.

      • lprent 1.1.2

        Just while I remember, the HTML ins and del need some css coloring and decoration. del should be coloured, and the insertions should be a different colour.

Recent Comments

Recent Posts

  • New Zealand expresses condolences on passing of Vanuatu High Commissioner
    Foreign Affairs Minister Winston Peters today extended New Zealand’s condolences following the death of Vanuatu’s High Commissioner to New Zealand, Johnson Naviti, who passed away yesterday afternoon in Wellington. “Our thoughts are with the High Commissioner’s family and colleagues during this difficult time. This is a terrible loss both to ...
    BeehiveBy beehive.govt.nz
    15 hours ago
  • Government announces allocation of three waters funds for councils
    The Government has today set out the regional allocations of the $761 million Three Waters stimulus and reform funding for councils announced by Prime Minister Hon Jacinda Ardern this month.  "I want to thank Councils around the country for engaging with the Central Local Government Steering Group who have been ...
    BeehiveBy beehive.govt.nz
    16 hours ago
  • Funding boost for students with highest learning support needs
    Students with high and complex learning needs, as well as their teachers and parents, will benefit from a substantial increase to Ongoing Resourcing Scheme (ORS) funding, Associate Education Minister Martin announced today. “Nearly $160 million will go towards helping these students by lifting their base support over the next four ...
    BeehiveBy beehive.govt.nz
    17 hours ago
  • Govt connecting kiwis to affordable, healthy food
    Funding for innovative projects to connect Kiwis with affordable, safe and wholesome food, reduce food waste, and help our food producers recover from COVID-19 has been announced today by Agriculture Minister Damien O’Connor. “COVID-19 has seen an increasing number of families facing unprecedented financial pressure. Foodbanks and community food service ...
    BeehiveBy beehive.govt.nz
    18 hours ago
  • Getting infrastructure for housing underway
    Eight shovel-ready projects within Kāinga Ora large-scale developments, and the Unitec residential development in Auckland have been given the go-ahead, Minister for Housing Dr Megan Woods announced today. Megan Woods says these significant infrastructure upgrades will ensure that the provision of homes in Auckland can continue apace. “The funding announced ...
    BeehiveBy beehive.govt.nz
    19 hours ago
  • Napier walk and cycleway to improve safety
    The Government is funding a new separated walking and cycleway path along Napier’s Chambers and Ellison streets to provide safer access for local students and residents across Marine Parade and State Highway 51, Transport Minister Phil Twyford and Police Minister Stuart Nash announced today. Funding of $2.7 million has been ...
    BeehiveBy beehive.govt.nz
    22 hours ago
  • PGF creates more than 10k jobs, success stories across NZ
    More than 13,000 people have been employed so far thanks to the Coalition Government’s Provincial Growth Fund, Regional Economic Development Minister Shane Jones has today announced. The number of jobs created by Provincial Growth Fund (PGF) investments has outstripped the 10,000 jobs target that the Government and Provincial Development Unit ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Inaugural seafood awards honour sustainability
    Scientists and innovative fishing operators from Stewart Island and Fiordland to Nelson, Marlborough and Wellington have been honoured in the first ever Seafood Sustainability Awards. Fisheries Minister Stuart Nash has congratulated the winners of the inaugural Seafood Sustainability Awards held at Parliament. “The awards night honours six winners, from a wide ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Climate resilience packages for regions
    The Government is providing an investment totalling more than $100 million for regions to protect against and mitigate the effects of climate change, Deputy Prime Minister Winston Peters and Infrastructure Minister Shane Jones have announced. Six regions will receive funding from the $3 billion allocated to infrastructure projects from the ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Southern Waikato shovel ready projects get the green light
    Three major local projects at Te Kuiti and Otorohanga have been given the money to get moving after the impact of Covid 19, says the Minister of Māori Development Hon Nanaia Mahuta.  The projects range from a Sports Centre for Te Kuiti, a redevelopment of the Otorohanga  Kiwi House and ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • New Zealand extends Middle East and Africa peace support deployments
    The Coalition Government has extended three New Zealand Defence Force deployments to the Middle East and Africa by two years, Foreign Affairs Minister Winston Peters and Defence Minister Ron Mark announced today.  “These deployments promote peace in the Middle East and Africa by protecting civilians and countering the spread of ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Govt progress on climate change essential, risk assessment shows
    The release of the National Climate Change Risk Assessment shows that the progress this Government has made to solve the climate crisis is essential to creating cleaner and safer communities across New Zealand. “Because of this report, we can see clearer than ever that the action our Government is taking ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • $10m sport recovery fund open for applications
    The second round of the Community Resilience Fund is now open for applications for sport and recreation organisations experiencing financial hardship between 1 July and 30 September 2020. “The fund opens today for five weeks – closing on September 6. The amount awarded will be decided on a case-by-case basis ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Rakitū Island declared latest predator free island
    Minister of Conservation Eugenie Sage today declared Rakitū Island, off the coast of Aotea/Great Barrier Island, predator free. “I’m delighted to announce that with rats now gone, Rakitū is officially predator free. This is a major milestone because Rakitū is the last DOC administered island in the Hauraki Gulf Marine ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Funding to restore significant Māori sites in the Far North
    The Provincial Growth Fund is investing $8.75 million to restore significant historic sites at Ōhaeawai in the Far North, upgrade marae and fund fencing and riparian planting. Regional Economic Development Minister Shane Jones made the announcements following a service at the historic St Michael’s Anglican Church at Ōhaeawai today.  Just ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Big boost for Chatham Islands’ economy
    The Chatham Islands will receive close to $40 million for projects that will improve its infrastructure, add to its attraction as a visitor destination, and create jobs through a planned aquaculture venture, Infrastructure Minister Shane Jones has announced. “The COVID-19 pandemic has had a devastating impact on the islands, first ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • More initiatives to reduce energy hardship
    The Government is delivering more initiatives to reduce energy hardship and to give small electricity consumers a voice, Energy and Resources Minister Megan Woods said today. “In addition to the initiatives we have already delivered to support New Zealand families, we are responding to the Electricity Price Review with further ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Turning the tide for hoiho/yellow-eyed penguin
    Government, iwi, NGOs and rehabilitation groups are working together to turn around the fortunes of the nationally endangered hoiho/yellow-eyed penguin following a series of terrible breeding seasons.  The Minister of Conservation Eugenie Sage helped launch the Five Year Action Plan at the annual Yellow-Eyed Penguin symposium in Dunedin today. “I ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Taskforce ready to tackle tourism challenges
    The membership of the Tourism Futures Taskforce has now been confirmed, Tourism Minister Kelvin Davis announced at an event at Whakarewarewa in Rotorua today. “The main purpose of the independent Tourism Futures Taskforce is to lead the thinking on the future of tourism in New Zealand,” Kelvin Davis said. Joining ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Investing in the tourism sector’s recovery
    More than $300 million in funding has been approved to protect strategic tourism businesses, drive domestic tourism through regional events and lift digital capability in the tourism industry, Tourism Minister Kelvin Davis announced today. A $400 million Tourism Recovery Package was announced at Budget 2020, and with today’s announcements is ...
    BeehiveBy beehive.govt.nz
    4 days ago
  • Permits to be required for exporting hard-to-recycle plastic waste
    From 2021 permits will be required for New Zealanders wanting to export hard-to-recycle plastic waste. The Associate Minister for the Environment, Eugenie Sage, today announced the requirements as part of New Zealand’s commitments to the Basel Convention, an international agreement of more than 180 countries which was amended in May ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Growth in new building consents shows demand is still high
    The building and construction sector is still showing strong growth, with the number of new dwellings consented up more than 8 per cent compared to last year, reflecting a welcome confidence in the Government’s COVID-19 response package, Minister for Building and Construction Jenny Salesa says. “While it is still too ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • $23 million for Bay of Plenty flood protection
    Government investment of $23 million for Bay of Plenty flood protection will allow local communities to address long-standing flood risks and provide jobs, Deputy Prime Minister Winston Peters and Under-Secretary for Regional Economic Development Fletcher Tabuteau announced in Rotorua today. These projects are being funded by the Infrastructure Reference Group’s (IRG) shovel ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Rotorua benefits from over $62 million boost
    Investment for projects that will create hundreds of jobs in Rotorua were announced today by Deputy Prime Minister Winston Peters and Under-Secretary for Regional Economic Development Fletcher Tabuteau. These projects will provide opportunities for economic development in a region that has been hard hit by COVID-19,” Winston Peters said. Fletcher ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Increased counselling support for all students
    For the first time, primary schools will have access to funding for counsellors for their students, Associate Education Minister Tracey Martin announced today. “A major investment of $75.8 million will provide greater access to guidance counsellors to help primary and secondary school students deal with mental health and wellbeing issues,” ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Report of the Government Inquiry into Operation Burnham released
    Defence Minister Ron Mark today welcomed the release of the Report of the Government Inquiry into Operation Burnham and related matters, and the Government response.  “I thank the Inquiry for their thorough and detailed report, on a highly complex issue. I accept the recommendations of the report, and fully support ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • 1BT funds create jobs and lasting benefits
    Forestry Minister Shane Jones has announced $6 million of One Billion Trees funding for seven regional initiatives to create jobs and provide long-lasting environmental and economic benefits. The projects range from improving one of the poorest-quality water catchments in Otago to restoring 52km of waterways around Hokianga Harbour. Six of the ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Kawerau projects to receive $5.5 million from Provincial Growth Fund
    Deputy Prime Minister Winston Peters today announced $5.5 million in funding from the Provincial Growth Fund (PGF) for two Kawerau projects and says this is a significant boost for the people of Kawerau. “These projects will bring much-needed investment and will create up to 60 jobs for locals,” Mr Peters ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • $5 million for Kaingaroa Village Redevelopment
    Kaingaroa Village in the Bay of Plenty is to get $5 million to help fund a comprehensive upgrade of its infrastructure, facilities and housing, Under-Secretary for Regional Economic Development Fletcher Tabuteau announced today. Mr Tabuteau travelled to the remote village to make the announcement, telling Kaingaroa residents how the funding ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • $18 Million Funding Boost for Bay of Plenty Business Park
    The Rangiuru Business Park project near Te Puke is getting $18 million from the Provincial Growth Fund (PGF), Under-Secretary for Regional Economic Development Fletcher Tabuteau announced today. “This is all about unlocking the potential of this region. When it’s finished, the Rangiuru Business Park will be the Bay of Plenty’s ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Town revitalisation and aquaculture investments create jobs in Ōpōtiki
    Deputy Prime Minister Winston Peters has today announced that a $26 million investment in Ōpōtiki will see important public amenities upgraded and further progress made on new aquaculture opportunities. “The people of Ōpōtiki have been waiting decades for real investment in key infrastructure, and support for the incredible aquaculture opportunities ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Minister congratulates the Cook Islands community for its 9th year of Language Weeks
    The Minister for Pacific Peoples, Aupito William Sio wishes to congratulate the Cook Islands community throughout Aotearoa for the 9th year of Te ‘Epetoma o Te Reo Māori Kūki ‘Āirani, the Cook Islands Language Week.  “This is a proud milestone that reflects on the huge effort made by the Cook ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Construction underway on longest section of Glen Innes to Tāmaki Drive Shared Path
    Aucklanders in the Eastern Suburbs will soon have more ways to get around, with Transport Minister Phil Twyford and Associate Transport Minister Julie Anne Genter kicking off construction on Section 2 of Te Ara Ki Uta Ki Tai, the Glen Innes to Tāmaki Drive Shared Path today. The Glen Innes ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • 350,000 More Measles Vaccines for Massive Immunisation Campaign
    The Government is stepping up the fight against measles and protecting hundreds of thousands more young adults by investing up to $40 million for a year-long measles-catch-up campaign and $23 million to fully fund and develop the National Immunisation Solution, Associate Minister of Health Julie Anne Genter announced at Mangere ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Operation Burnham report released
    Attorney-General David Parker has today released the findings of the Government inquiry held into Operation Burnham and related events. The operation took place on 21-22 August 2010 in Tirgiran Valley, Afghanistan, and was carried out by NZSAS troops and other nations’ forces operating as part of the International Security Assistance ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Locally-led solutions at centre of new community resilience fund
    From tomorrow, community groups around New Zealand can apply to a $36 million fund established to encourage locally-led solutions as communities rebuild and recover from COVID-19, announced Minister for Social Development Carmel Sepuloni and Minister for Community and Voluntary Sector Poto Williams. “The Community Capability and Resilience Fund (CCRF) builds ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Securing healthy futures for all Māori
    The Government has committed to improving Māori health and wellbeing over the next five years. The Associate Minister of Health (Māori Health) today released Whakamaua: Māori Health Action Plan 2020-2025 which sets the pathway towards achieving healthy futures for all Māori. “As kaitiaki of the system, the Ministry of Health ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Porirua Development delivers more new public housing
    The first of nearly 70 new state homes have been opened in Cannons Creek, Porirua by the Associate Minister of Housing, Kris Faafoi, as part of an increase in public housing being delivered through the Porirua Development.  “Completion of the first 10 of 53 new two and five bedroom homes ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • New standards for existing marine farms provide consistency
    New environmental standards will make the re-consenting of existing marine farms more consistent across the country.  The new regulations for the National Environmental Standards for Marine Aquaculture (NES-MA) will come into effect on 1 December, Environment Minister David Parker and Fisheries Minister Stuart Nash said.   “The NES-MA removes complexities and ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Government signs Accord reinvigorating commitment to Far North iwi
    Today marks a milestone as the Minister for Social Development Carmel Sepuloni and Minister for Māori Development Nanaia Mahuta co-sign an Addendum – with the Iwi Chairs of Te Rarawa, Ngāi Takoto and Te Aupōuri – to the Te Hiku o Te Ika Iwi-Crown Social Development and Wellbeing Accord (the ...
    BeehiveBy beehive.govt.nz
    6 days ago