web analytics

Search upgraded (yet again)

Written By: - Date published: 12:47 am, March 15th, 2010 - 4 comments
Categories: admin - Tags:

My absence this weekend from moderation and posts has partially been because of going to the excellent Fabian seminar today. However it has largely been due to trying to get a workable search system running.

One that didn’t bring the whole server crashing around my ears every few months and cut the issue of everyone having to stop receiving data for up to a minute when someone used search. One that allowed comments to be accessed as easily as posts, which Search Unleashed achieved (which eventually failed because the re-index took too long). It looks like it is finally working correctly, but I’d like feedback on any errors that show up.

For the technically minded, this is running on the Sphinx search engine with the WordPress Sphinx Search Plugin, and some custom coding in the search sidebar and search page template for our new theme. It runs a delta update for changed items every 5 minutes, and a complete rebuild at 0300.

This is the third time I’ve attempted to get this system running. Each of the previous attempts have been stymied by sphinx disliking our constrained memory during peak hours. But we’ve bumped the RAM up, and it also turns out that there were some conflicting permission issues (grrr) when running the cron jobs with an active website.

Normally I’d have consigned something as annoying to configure as this to the dustbin, but the engine is blindingly fast and perfect for our scaling needs with nearly 5000 posts and 150,000 comments currently onsite. This is expected to keep rising at rates similar or higher than have been the case so far.

Sphinx can reindex our entire database in about 30 seconds (whereas Search Unleashed required hours), and will do the delta every 5 minutes in well under a second. Almost all of the delay in presenting the search result is from assembling the display rather than the search. It is a seriously good engine and has been (so far) worth the effort.

The user interface is a little primitive at present and I also need to store and get your user selected advanced settings in your cookies. But it is good enough to look for loading problems over this week. It has a number of options that will be useful for advanced searching, especially for authors like r0b…

Sphinx extended queries syntax allows the following special operators to be used:

  • AND: hello&world hello world
  • OR: hello | world
  • NOT: hello -world or hello !world
  • field search operator: @title hello @body world
  • phrase search operator: “hello world”
  • proximity search operator: “hello world”~10

Here’s an example query which uses all these operators:
“hello world” @title “example program”~5 @body python -(php|perl)

More information about extended syntax you can find at Sphinx.

The following field operators are available:

  • @title – search in post or page title
  • @author – search in post, page, comment author (still a bit flakey)
  • @body – search in post, page, comment body
  • @category – search search in blog categories

For instance, I tested such things as lprent&fabian lprent fabian, pigs-flying, and (supershitty|super-city|super city|supercity)& franklin.

Using the author you can search for f*ck @author robinsod (that allows you to exclude the people swearing at him).

If you use the advanced and turn pages and comments off, you can resolve a long-standing argument.

  • @title “helen clark” and @title “john key” will respectively tell you how often each was mentioned in the title of a post.
  • @body “helen clark” and @body “john key” will respectively tell you how often each was mentioned in the body of a post.

If I see that argument again…..

The time to run a search seems to be roughly the same regardless of how complex I make the query.

Anyway, let me know if you find any issues using the form in contact. I’ll be keeping an eye on it for the next 3 or 4 days for performance issues.

Hopefully I’ll now have more time  to write posts and tackle the minor teething issues like the misbehaving Contribute Past captcha, and less time trying to maintain a fragile search mechanism that was slowly fading under increased loads.

BTW: For those of you who were at the Fabian seminar today. I was paying attention to the panel and discussion. But I was also coding during the seminar. This had been bugging me most of the weekend to the point where I had to go to the seminar, tether the iPhone to the net, and finish the damn search page interface.

Update: Been through the site settings for the first time since it last had a server move. You’ll see a significant improvement in performance in the search, and probably in just reading the site.

Update: Found a couple of errors in the post, which you will see corrected with the original in strikeout. There are a number of enhancements to the system (like @author). I added some additional examples while testing.

But I was paying attention to the seminar as well

4 comments on “Search upgraded (yet again) ”

  1. lprent 1

    Ummm, one thing it misses is the comment author name…. and post author name..

    Ok that is a problem in the query, which I’ll adjust in the sphinx.conf tomorrow sometime

    • mummybot 1.1

      Good on ya mate! 😉

      • lprent 1.1.1

        Did a burst of database optimization tonight as well. I’m pretty sure that the site will be significantly faster for reading and probably the same or slightly better for writing.

        Sphinx was loading down the system (as well as some of the other queries) so I tuned up a couple of tables with indexes based on what was showing in the slow query log. Then I tuned mysql to use more of the memory/disk/cpu tradeoff. Then did some tweaks to the php settings based on its logs.

        I’d better give up some of these late night sessions.

      • lprent 1.1.2

        Just while I remember, the HTML ins and del need some css coloring and decoration. del should be coloured, and the insertions should be a different colour.

Recent Comments

Recent Posts

  • Keynote address to Chartered Accountants Australia and New Zealand conference
    Speech to the CAANZ conference - November 19, 2020 Thank you, Greg, (Greg Haddon, MC) for the welcome. I’d like to acknowledge John Cuthbertson from CAANZ, the Commissioner of Inland Revenue Naomi Ferguson, former fellow MP and former Minister of Revenue, Peter Dunne, other guest speakers and CAANZ members. I ...
    BeehiveBy beehive.govt.nz
    22 hours ago
  • Expert independent advisory group appointed to strengthen the future of Māori broadcasting
    A panel of seven experts are adding their support to help shape the future of Māori broadcasting, Minister for Māori Development Willie Jackson has announced today. “Today I will meet with some of the most experienced Māori broadcasters, commentators and practitioners in the field. They have practical insights on the ...
    BeehiveBy beehive.govt.nz
    1 day ago
  • Government to review housing settings
    New Zealand’s stronger-than-expected economic performance has flowed through to housing demand, so the Government will review housing settings to improve access to the market, the Finance Minister Grant Robertson announced today. “Our focus is on improving access to the housing market for first home buyers and ensuring house price growth ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Crown accounts reflect Govt’s careful economic management
    The better-than-expected Crown accounts released today show the Government’s careful management of the COVID-19 health crisis was the right approach to support the economy. As expected, the Crown accounts for the year to June 2020 show the operating balance before gains and losses, or OBEGAL, was in deficit. However that ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Community launch marks next step in addressing racism in education
    The launch of Te Hurihanganui in Porirua today is another important milestone in the work needed to address racism in the education system and improve outcomes for Māori learners and their whānau, Associate Education Minister Kelvin Davis says. Budget 2019 included $42 million over three years to put Te Hurihanganui ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Government to consider recommendations on DNA use in criminal investigations
    The Minister of Justice has received the Law Commission’s recommending changes to the law governing the way DNA is used in criminal investigations. The report, called The Use of DNA in Criminal Investigations – Te Whahamahi I te Ira Tangata I ngā Mātai Taihara, recommends new legislation to address how ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Speech to Wakatū Nelson regional hui on trade
    First, I want to express my thanks to Te Taumata for this hui and for all the fantastic work you are doing for Māori in the trade space. In the short time that you’ve been operating you’ve already contributed an enormous amount to the conversation, and developed impressive networks.  I ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Speech to Primary Industries Summit
    Thank you for the opportunity to speak to you today about the significant contribution the food and fibres sector makes to New Zealand and how this Government is supporting that effort. I’d like to start by acknowledging our co-Chairs, Terry Copeland and Mavis Mullins, my colleague, Agriculture Minister Damien O’Connor, ...
    BeehiveBy beehive.govt.nz
    2 days ago
  • Fast track referrals will speed up recovery and boost jobs and home building
    The Government is taking action to increase jobs, speed up the economic recovery and build houses by putting three more projects through its fast track approval process. “It’s great to see that the fast-track consenting process is working. Today we have referred a mix of potential projects that, if approved, ...
    BeehiveBy beehive.govt.nz
    3 days ago
  • Papakāinga provides critically needed homes in Hastings
    A papakāinga opened today by the Minister for Māori Development the Hon Willie Jackson will provide whānau with much needed affordable rental homes in Hastings. The four home papakāinga in Waiōhiki is the first project to be completed under the ‘Hastings Place Based’ initiative. This initiative is a Government, Hastings ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • New Zealand ready to host APEC virtually
    Prime Minister Jacinda Ardern took over the leadership of APEC earlier today, when she joined leaders from the 21 APEC economies virtually for the forum’s final 2020 meeting. “We look forward to hosting a fully virtual APEC 2021 next year. While this isn’t an in-person meeting, it will be one ...
    BeehiveBy beehive.govt.nz
    5 days ago
  • Revival of Māori Horticulturists
    The rapid revival of Māori horticulture was unmistakeable at this year’s Ahuwhenua Trophy Awards, with 2020 marking the first time this iconic Māori farming event was dedicated to horticulture enterprises. Congratulating finalists at the Awards, Māori Development Minister Willie Jackson said growing large-scale māra kai is part of Māori DNA. ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Emergency benefit to help temporary visa holders
    From 1 December, people on temporary work, student or visitor visas who can’t return home and or support themselves may get an Emergency Benefit from the Ministry of Social Development, Social Development and Employment Minister Carmel Sepuloni announced today. Previously, temporary visa holders in hardship because of COVID-19 have had ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • School sustainability projects to help boost regional economies
    Forty one schools from the Far North to Southland will receive funding for projects that will reduce schools’ emissions and save them money, Education Minister Chris Hipkins announced today. This is the second round of the Sustainability Contestable Fund, and work will begin immediately. The first round announced in April ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Farmer-led projects to improve water health in Canterbury and Otago
    More than $6 million will be spent on helping farmers improve the health of rivers, wetlands, and habitat biodiversity in Canterbury and Otago, as well as improving long-term land management practices, says Agriculture Minister Damien O’Connor. Four farmer-led catchment group Jobs for Nature projects have between allocated between $176,000 and ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Tupu Aotearoa continues expansion to Pacific communities in Nelson, Marlborough, Tasman & Northl...
    Pacific communities in Nelson, Marlborough, Tasman and Northland will benefit from the expansion of the Tupu Aotearoa programme announced today by the Minister for Pacific Peoples, Aupito William Sio. The programme provides sustainable employment and education pathways and will be delivered in partnership with three providers in Northland and two ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • New primary school and classrooms for 1,200 students in South Island
    Education Minister Chris Hipkins unveiled major school building projects across the South Island during a visit to Waimea College in Nelson today. It’s part of the Government’s latest investment of $164 million to build new classrooms and upgrade schools around the country. “Investments like this gives the construction industry certainty ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Minister of Māori Development pays tribute to Rudy Taylor
      Today the Minister of Māori Development, alongside other Government Ministers and MP’s said their final farewells to Nga Puhi Leader Rudy Taylor.  “Rudy dedicated his life to the betterment of Māori, and his strong approach was always from the ground up, grassroots, sincere and unfaltering”  “Over the past few ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Prime Minister to attend APEC Leaders’ Summit
    Prime Minister Jacinda Ardern will attend the annual APEC Economic Leaders’ Meeting and associated events virtually today and tomorrow. “In a world where we cannot travel due to COVID-19, continuing close collaboration with our regional partners is key to accelerating New Zealand’s economic recovery,” Jacinda Ardern said. “There is wide ...
    BeehiveBy beehive.govt.nz
    6 days ago
  • Speech to Infrastructure NZ Symposium
    Tena Koutou, Tena Koutou and thank you for inviting me to speak to you today. This is a critical time for New Zealand as we respond to the damage wreaked by the global COVID-19 pandemic. It is vital that investment in our economic recovery is well thought through, and makes ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Pike River 10 Year Anniversary Commemorative Service
    Tēnei te mihi ki a tātau katoa e huihui nei i tēnei rā Ki a koutou ngā whānau o te hunga kua riro i kōnei – he mihi aroha ki a koutou Ki te hapori whānui – tēnā koutou Ki ngā tāngata whenua – tēnā koutou Ki ngā mate, e ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Huge investment in new and upgraded classrooms to boost construction jobs
    Around 7,500 students are set to benefit from the Government’s latest investment of $164 million to build new classrooms and upgrade schools around the country. “The election delivered a clear mandate to accelerate our economic recovery and build back better. That’s why we are prioritising construction projects in schools so more ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Keeping Pike River Mine promises 10 years on
    Ten years after the Pike River Mine tragedy in which 29 men lost their lives while at work, a commemorative service at Parliament has honoured them and their legacy of ensuring all New Zealand workplaces are safe. Prime Minister Jacinda Ardern attended the event, along with representatives of the Pike ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • Additional testing to strengthen border and increase safety of workers
    New testing measures are being put in place to increase the safety of border workers and further strengthen New Zealand’s barriers against COVID-19, COVID-19 Response Minister Chris Hipkins said today. “These strengthened rules – to apply to all international airports and ports – build on the mandatory testing orders we’ve ...
    BeehiveBy beehive.govt.nz
    7 days ago
  • More public housing delivered in Auckland
    The Government’s investment in public housing is delivering more warm, dry homes with today’s official opening of 82 new apartments in New Lynn by the Housing Minister Megan Woods. The Thom Street development replaces 16 houses built in the 1940s, with brand new fit-for-purpose public housing that is in high ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Agreement advanced to purchase up to 5 million COVID-19 vaccines
    The Government has confirmed an in-principle agreement to purchase up to 5 million COVID-19 vaccines – enough for 5 million people – from Janssen Pharmaceutica, subject to the vaccine successfully completing clinical trials and passing regulatory approvals in New Zealand, says Research, Science and Innovation Minister Megan Woods. “This agreement ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Jobs for Nature funding will leave a conservation legacy for Waikanae awa
    Ninety-two jobs will be created to help environmental restoration in the Waikanae River catchment through $8.5 million of Jobs for Nature funding, Conservation Minister Kiritapu Allan announced today. “The new funding will give a four-year boost to the restoration of the Waikanae awa, and is specifically focussed on restoration through ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Dunedin Hospital project progresses to next stage
    As the new Dunedin Hospital project progresses, the Government is changing the oversight group to provide more technical input, ensure continued local representation, and to make sure lessons learnt from Dunedin benefit other health infrastructure projects around the country. Concept design approval and the release of a tender for early ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Jump in apprentice and trainee numbers
    The number of New Zealanders taking up apprenticeships has increased nearly 50 percent, and the number of female apprentices has more than doubled. This comes as a Government campaign to raise the profile of vocational education and training (VET) begins. Prime Minister Jacinda Ardern and Education Minister Chris Hipkins announced ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • ReBuilding Nations Symposium 2020 (Infrastructure NZ Conference opening session)
    Tena koutou katoa and thank you for the opportunity to be with you today. Can I acknowledge Ngarimu Blair, Ngati Whatua, and Mayor Phil Goff for the welcome. Before I start with my substantive comments, I do want to acknowledge the hard work it has taken by everyone to ensure ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Zealand's biosecurity champions honoured
    Biosecurity Minister Damien O’Connor has paid tribute to the winners of the 2020 New Zealand Biosecurity Awards. “These are the people and organisations who go above and beyond to protect Aotearoa from pests and disease to ensure our unique way of life is sustained for future generations,” Damien O’Connor says. ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Tourism Industry Aotearoa Conference
    speech to Tourism Industry Aotearoa annual summit Te Papa,  Wellington Introduction Nau mai, haere mai Tēnā koutou, tēnā koutou, Ka nui te mihi, ki a koutou. Thank you Tourism Industry Aotearoa for hosting today’s Summit. In particular, my acknowledgements to TIA Chair Gráinne Troute and Chief Executive Chris Roberts. You ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Supermarkets announced as Government’s second market study
    The Government has today launched a market study to ensure New Zealanders are paying a fair price for groceries.   “Supermarkets are an integral part of our communities and economy, so it’s important to ensure that Kiwis are getting a fair deal at the checkout,” Minister of Commerce and Consumer ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • Masks to be worn on Auckland public transport and all domestic flights
    Masks will need to be worn on all public transport in Auckland and in and out of Auckland and on domestic flights throughout the country from this Thursday, Minister for COVID-19 Response Chris Hipkins said today. “I will be issuing an Order under the COVID-19 Response Act requiring the wearing ...
    BeehiveBy beehive.govt.nz
    1 week ago
  • New Zealand signs Regional Comprehensive Economic Partnership
    Increase to New Zealand’s GDP by around $2 billion each year Increase opportunities for NZ exporters to access regional markets Cuts red tape and offers one set of trade rules across the Asia Pacific region New government procurement, competition policy and electronic commerce offers NZ exporters increased business opportunities Prime ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Minister acknowledges students as exams begin
    Education Minister Chris Hipkins has recognised the extraordinary challenges students have faced this year, ahead of NCEA and New Zealand Scholarship exams which begin on Monday. “I want to congratulate students for their hard work during a year of unprecedented disruption, and I wish students all the best as ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Prime Minister meets with key ASEAN and East Asia Summit partners
    Prime Minister Jacinda Ardern today attended the ASEAN-New Zealand Commemorative Summit and discussed with Leaders a range of shared challenges facing the Indo-Pacific region, including: The ongoing management of the COVID-19 pandemic; The importance of working collectively to accelerate economic recovery; and Exploring further opportunities for partners to work more ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Veterans Affairs Summit held in Korea
    A Ministerial Summit on Veterans’ Affairs was held in the Republic of Korea this week. Ministers with veteran responsibilities were invited from all 22 countries that had been part of the United Nations Forces during the Korean War (1950 – 1953). The Summit marked the 70th anniversary of the outbreak ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • Clear direction set for the education system, skills prioritised
    The Government has released a set of priorities for early learning through to tertiary education and lifelong learning to build a stronger, fairer education system that delivers for all New Zealanders. “The election delivered a clear mandate from New Zealanders to accelerate our plan to reduce inequalities and make more ...
    BeehiveBy beehive.govt.nz
    2 weeks ago
  • A Progressive Agenda
    Speech to the Climate Change + Business Conference, November 12, 2020 Tena koutou katoa Thank you for inviting me to speak here today. It is great to see us all come together for a common cause: to redefine our future in the face of unprecedented times.  Covid-19 and climate change are ...
    BeehiveBy beehive.govt.nz
    2 weeks ago