Talk:Opinion polling on Scottish independence

FindOutNow

I've taken a look at the many, many polls conducted in 2023, and there is only a single poll not conducted by FindOutNow that has shown a Yes result. A single poll out of dozens and dozens. FindOutNow seems to consistently be inflating the Yes result in its polls. The impartiality of the pollsters has to be called into question too considering they are quite clearly pro-separatism and somehow seem to keep churning out polls that are completely at odds with every other mainstream poll being conducted around the same time. 2.99.68.104 (talk) 04:49, 8 July 2023 (UTC) BE sock. Mutt Lunker (talk) 12:37, 5 August 2023 (UTC)

No, none of the polling companies listed here (which are all members of the BPC or abide by their rules) are biased or in favour of one side or the other. Any accusation of impartiality by any of the pollsters listed here is false. It serves them no benefit.

The methodologies of pollsters do produce different results. Most appear to be pretty neutral but Find Out Now do produce results with the most extreme variation from the mean of any pollster. Their method for getting a sample is very different to other pollsters and they are also much cheaper which has the potential for more ‘exploitation’. For example, there is currently a lot of chat in the pro-independence blogosphere about how FON dont weight for 2014 (which it’s argued is 9 years out of date). However it conveniently ignores the fact that Redfield & Wilton don’t weight for 2014 either and they seem to produce results favouring No. For example this twitter post by someone who actually commissioned a FON poll which appears to me to be factually incorrect: https://twitter.com/Celebs4indy/status/1677264079818194945. The point here is that the poll company itself is not inherently biased, but that their results can be used in a biased way. All this has been discussed previously on this page-perhaps you should read this before commenting? Dunk the Lunk (talk) 06:31, 8 July 2023 (UTC)

"Find Out Now do produce results with the most extreme variation from the mean of any pollster."

And they also happen to be openly pro-separatist and thus biased. You don't find that strange at all. That the pro-separatist outlet just happens to keep churning out polls showing support for the cause it supports a good 10 points higher on average than every other poll conducted around the same timeframe.

If there was an outlet that was openly British nationalist and it kept showing polls showing support for remaining a part of Britain a good 10 points higher than every other poll I'd be raising the same questions.

Something tells me you'd be agreeing with me in that situation, though. So why the ridiculousness here? 2.99.68.104 (talk) 09:39, 8 July 2023 (UTC)

Oh wait you do largely agree with me, I went back and read what you suggested I read. Oh well, nevermind then. Guess there's nothing anyone can do until the BPC excludes them or blacklists them or whatever. Ridiculous it's gone on for this long. 2.99.68.104 (talk) 09:55, 8 July 2023 (UTC) BE sock. Mutt Lunker (talk) 12:37, 5 August 2023 (UTC)

I completely agree with Dunk the Lunk

We have had this discussion and we seem to agree that it is methodology bias rather than any company deliberately bias. Soosider3 (talk) 10:28, 8 July 2023 (UTC)

Oh. Oh hold on. For some reason I was of the opinion that Find Out Now was in some way affiliated with or funded by The National. I was of the impression it was a polling organization working for a pro-separatist media outlet. I appear to be gravely mistaken with that belief.

Alright, well I firmly retract my claims about deliberate bias on the part of the pollsters then, but continue to echo the complaints about methodology bias. 2.99.68.104 (talk) 10:54, 8 July 2023 (UTC) BE sock. Mutt Lunker (talk) 12:37, 5 August 2023 (UTC)

I don't think FON are pro Indy. The trouble is their method is deeply flawed for a controversial issue with thousands of committed activists in the population. Consider ALBA is an organisation whose sole purpose, leaving aside Alex Salmond's ego, is to game the electoral system to inflate pro-Independence representation.

So consider ALBA commission an Indyref Poll from FON. Is it so far beyond imagining that some ALBA supporters (are encouraged to) flood the FON website in the relevant window to distort the poll? I can tell you what lies they would have to tell, but you can probably work it out.

The method is not fit for purpose in this context.

I think we have to carry BPC polls though. It might make sense to strengthen the leading comments about differences in methodology affecting results.

I've written to the BPC about this in general terms, to no response so far. I will do anything else I can think of to get some sensible changes made, which admittedly might not be much. RERTwiki (talk) 16:54, 12 July 2023 (UTC)

Can I suggest we take some time out here, we seem to be in danger of lapsing into some of the less desirable behaviours of social media, with accusations of bias etc etc. We edit a poll aggregator and generally despite the different stances we might have on Independence, we do so in a balanced and consistent manner. Lets not let the toxicity that can exist on the topic, in other places, pollute our efforts here.

Judging by the views this article gets, we seem to have some value for readers, I believe that is because our present system is fairly robust and simple, if pollster is part of BPC we record it and note any variations. As we are more immersed than many in this perhaps we are more aware of Methodology variations that different pollsters have, pollsters are not biased. Suggest we use that phrase "methodology variations" rather than bias - it has too many connotations from other places. Soosider3 (talk) 20:49, 12 July 2023 (UTC)

If I might add something here. I manage the Independent Voices blog and have commissioned two polls from Find Out Now in 2023. They are professional, dilligent and strong on communication around method. Everything is transparent. There is no value in trying to influence polling companies one way or another, FON just happen to be inexpensive and accessible. The big factor in the polling company differences is pretty simple: it's how the sample is constructed. Polls that use the 2014 vote to weight their samples tend to boost the number of No voters in that sample. The No voting intention is strongly correlated to the % of No voters in the sample. Some pollsters have samples that are a poor reflection of the current electorate, for example Panelbase have only 12% who did not vote in 2014 - this figure realistically should be around 27%. Pollsters that have a decent % of new voters and that don't boost the No sample by 2014 weighting tend to show Yes in the lead. Not just Ipsos, also Opinium, Hanbury and FON. Oh, and both Yougov (December 22) and Savanta (Feb 21) have published polls unweighted by 2014 result (both with Yes leads). Given that these polls are funded by the Times and Scotsman unsurprising that they went back to their 2014 weighting! IndyVoices (talk) 15:34, 4 December 2023 (UTC)

I agree with you exactly on the actions we should take: keep tracking all polls to keep people informed.

I'm not accusing any pollster of bias. I am saying that I think the FON method has obvious vulnerabilities which makes it unsuitable for independence polls, though it nonetheless has to be included until the BPC takes action. I don't know enough about other methods to comment on them.

I do think the leading caveat on methodology needs to be strengthened, and will float something below. RERTwiki (talk) 16:35, 13 July 2023 (UTC)

The accusations of bias were on my part and it was due to a misreading/misunderstanding of the data. I thought FON was affiliated with pro-separatist organizations due to the fact they were running the poll in that outlet or something. My mistake. 2A00:23EE:15E8:2A84:7995:D699:FAEA:EAC7 (talk) 19:56, 17 July 2023 (UTC) BE sock. Mutt Lunker (talk) 12:37, 5 August 2023 (UTC)

I have been thinking about the whole postcode lottery/find out now issue. One thing I don’t understand is some (not all) of their polls include 16-17 year olds. How does this work when you have to be 18 to register on their site? Or am I missing something? Dunk the Lunk (talk) 17:50, 29 July 2023 (UTC)

Find Out Now Sep 8-13 Poll

I have added link to data table, however I think we need to have a discussion as to which data table we should link to, FoN at the request of their client, has produced tables that have data on both weighing for or without 2014 referendum vote, it produces a small difference. However they have gone on and ascertained Voting intention, likelihood to vote and turnout weighing as well. These do produce a much larger change in projected outcomes, depending on which combination of weighing is used. If folk could have a look at the 8 data sets produces and share thoughts as to which we should use for main table. I tend to think that the first 2 tables offered are the ones most consistent with other pollsters, however if other weights used were to become more commonly used then we may need to open a sub table to capture them Have a look and let me know what you think https://findoutnow.co.uk/blog/small-lead-for-yes-to-scottish-independence/ Soosider3 (talk) 10:31, 15 September 2023 (UTC)

Hmm it’s a tricky one I’ll agree. Generally I think we should report the headline figures in the main table but in this case FON appear to be reporting two headlines (‘With 2014 vote quotas’ and ‘Without 2014 vote quotas’). Obviously there needs to be the minimum of a note to state there are multiple versions weighted differently. I’m not sure which of the two headline ones would be best to be honest but happy to go with the consensus (unhelpful I know). However, I have to say this just increases my scepticism about the FON polls. It’s like they’re not confident in their own methodology so are throwing a load out and hoping one sticks…

With regard to the notes for 16/17 y/o and 2014 has this been done/checked throughout for consistency? Dunk the Lunk (talk) 15:14, 15 September 2023 (UTC)

I'm fairly sure the 16/17 year olds is highlighted where appropriate, not so sure with 2014. Soosider3 (talk) 16:11, 15 September 2023 (UTC)

Okay, I have checked all polls recorded in 2023 for consistency of weighing, done on a wee spreadsheet that I would be happy to share. Having looked at that I went back and changed the data link for the last FoN poll, reason for that was they had produced 8 different data tables, counting out and in various weighing, they had introduced a weighing that was new to me 'Turnout' this was in addition to 'Voting Intention' Having checked 2023 polls this is first time I can see such a weighing being used so felt it best avoid it. We can go back and look at it again if it becomes a regular occurrence, but for now its a one off and best avoided.

As a summary so far in 2023 there have been 35 Polls recorded that have access to full data tables of these

35 have weighed to 2019 GE

23 to 2021 SGE

25 to 2016 EU Ref

23 to 2014 Ref

28 for voting Intention

29 Include 16+

There is an unclear picture but think that at least some of the variations/trends/changes may be down as much to weighing variations as any thing else. Soosider3 (talk) 07:50, 16 September 2023 (UTC)

This makes sense and I’m happy with it as long as you/others are. I do think there should be a note with this rationale explained, potentially with a link to the FON website page you link too above? Dunk the Lunk (talk) 11:33, 16 September 2023 (UTC)

I'm a little confused as to why you would take out turnout weighting where it's available. Yougov are the only main pollster I know that don't weight for turnout in indyref polls, whcih only serves to give a large number of 'don't knows'. The likelihood to vote figures (as with everything) are transparent and available for scrutiny. The figures align very closely with the actual indyref turnout figure. Plus, I had to pay for the voting likelihood question! IndyVoices (talk) 15:44, 4 December 2023 (UTC)

FocalData Poll October

Was doing some tidy up work on links to data, really struggling to find data tables from this Polling Company, yes I can find published data from the client but we should be linking to best data. Focaldata appear to require an account to be able to access their published polls, which seemed a bit off. Anyone else having any luck with this? Soosider3 (talk) 11:59, 27 November 2023 (UTC)

Looking at the data published on the clients site and looking at the partial tables they published, it is difficult to see where the numbers for both the Yes/No question and the Leave/Remain, they appear to be subsets of about 500 responses and not the 1000+ claimed, this also raises the issue of reporting on what are actually subsets of a full poll.

Information can be found here

clients article https://www.these-islands.co.uk/publications/i389/its_the_way_you_ask_them.aspx

clients published tables on google docs https://docs.google.com/spreadsheets/d/1kkmgMPSSDAOfIgwjowGKJr6aXjowJaGc6fnZaP_O_IE/edit#gid=0

Please have a look and see what you think, to my mind there is something not right here, 4 weeks later and no pollster published data, sub sets masquerading as full reports. Soosider3 (talk) 14:20, 27 November 2023 (UTC)

I had emailed focaldata and got response pointing me to the google docs link, is this normal for a pollster? Have responded asking for more clarification especially around querying if this is their usual practice and why so many questions appear to be missing from the published information. Soosider3 (talk) 19:21, 27 November 2023 (UTC)

That's a couple of days now and no comment, should I take it that silence is agreement and remove this polling data? Soosider3 (talk) 14:43, 29 November 2023 (UTC)

Focaldata are a member of the BPC so their polling should be included in the article. One of the stipulations of being a member of the BPC is that the polling data should be published within 2 working days on their website. AlloDoon (talk) 10:21, 6 December 2023 (UTC)

Good morning, still not able to find data on their website. Any thoughts on data as published by client. These appear to be subsets but unlike previous smaller sample polls no indication that they have weighed for smaller sample, if correct this has to raise questions about the inclusion of this particular poll Soosider3 (talk) 10:39, 6 December 2023 (UTC)

It is acceptable for Focaldata to link to Google docs so long as it contains relevant polling data and sub-sampling. The weighted sample of the poll which you have linked on the independence question is 545 for both Yes/No and Remain/Leave questions. As this sample is weighted and within an appropriate margin of error below 7% it is acceptable to the poll listed in the article with a note explaining the sample size is smaller than 1,000 and has a margin of error above 3% AlloDoon (talk) 16:01, 6 December 2023 (UTC)

The point I was attempting to make was, if the link is not from the pollsters website can we be certain of its veracity, at the very least it raises the suspicion of a doubt. BPC rules very specific re publishing on pollster website. Until now all links to data tables have been from pollsters website and thereby can be assumed that they are the tables the pollsters stand by.

Re numbers of samples, not convinced that these are other than subsets of a full poll, I see no evidence of them weighting for smaller sample or generating what margin of error would now be. Its a question not a statement I am making, where is the evidence to support them being treated as proper polls, all be it small ones. Soosider3 (talk) 11:44, 7 December 2023 (UTC)

It getting on 2 weeks since I first posted my queries about this poll, in particular issue of link to data tables and use of subsets.

I am minded to remove them until we can establish there usability Soosider3 (talk) 10:16, 10 December 2023 (UTC)

This from 'theseislands' article is an amendment from the original publication, it offers an explanation as to how the numbers were arrived at, in essence they ask the same question twice at different parts of the interview then combine those numbers. Once eraly on and again later in process, there is no publication of all the questions between those referred to. This is unacceptable, as I am sure you realise that questioning order can and does have an impact on returns, to then merge these 2 separate and different responses to masquerade as a single questioning is highly dubious and to present as one response is misleading. I intend to delete Focaldata information until those who might support there inclusion can answer those queries. Soosider3 (talk) 13:42, 14 December 2023 (UTC)

I don't accept this at all, they are BPC members, not you. I'm going to revert these changes.

Your incessant determination to remove any poll which doesn't support your political agenda is abusive in the extreme, and needs to stop. RERTwiki (talk) 16:44, 14 December 2023 (UTC)

The assertion that about my motives do you no credit. Look at the data posted and try to square it with what was published by the client. The client stated in their article that they had asked the question twice as q6 and again as q24, effectively creating 2 subsets and then amalgamating them, at best highly questionable activity. I first raised my doubts over 2 weeks ago, but it has taken my action to garner a response. At present the pollster has not yet published the data on their website ( contrary to box rules) By all means explain if my rationale is faulty, I have always understood we don't publish subsets adding 2 together does not make it okay. I would remind you that not only the wording of questions but the order they are asked can impact results Soosider3 (talk) 17:09, 14 December 2023 (UTC)

The procedure specified in the InTheseIslands article is fine. They asked everyone both questions, and in order to avoid the bias implicit in, say, asking yes/no first and remain/leave second, they split the population into two and asked each in different orders. This seems eminently sensible to me, and obviously they have thought about the problem. They are the professionals here.

One can make a reasonable argument that visibility of both yes/no and remain/leave framing is what people would see in an actual campaign. For certain there will be a campaign for "Yes!" and the unionists would have to be blind and stupid not to campaign on "Remain". That is, asking both questions plausibly creates a more realistic response.

We are not quality control for the polling industry. This poll is good enough for What Scotland Thinks, and is produced by a BPC member. We should have a strong bias in favour of inclusion, for reasons previously cited. The poll should stay in the list.

Yes, the poll favours NO, but by about the same margin that MORI always favours YES and much less than the outlandish margins FindOutNow has favoured YES in some polls.

I don't appear to have the same conception of credit as you. As my grandmother used to say, least said soonest mended. RERTwiki (talk) 10:21, 15 December 2023 (UTC)

Your grandmother sounds a very wise person.

I would respond by pointing out that it is not the pollster offering these explanations but rather the client, that the pollster has not issue these tables (as of yet) and there is no explanation of methodology from the pollster. We should always be cautious about sources of information and how much weight to give them and would suggest that a campaign group maybe somewhat less reliable than the actual pollsters ( who are missing from this discussion)

Please note that it is well established in polling circles that when the question is asked in the polling questionnaire can impact its result, just as the questions asked leading to the question can also impact the result, we have no data as to what Questions 11 to 24 might have been.

As I say there is some highly questionable stuff happening here, as well as not polling 16/17 years old or weighing for likelihood to vote, we have (to the best of my knowledge) the Q being asked twice at different stages of questionnaire.

This is unusual practice and should be noted as such, perhaps we should include a separate line for each of the small polls with the appropriate information in the notes field, as if not subsets they are small polls

Notable that yet again we have a mention of the "methodology bias" we had some discussion on here earlier in the year around Findoutnow poll, I had did a wee bit of research on that and think there maybe a link between what pollsters weigh for in previous voting (ie Indyref, Brexit, Last GE and last Holyrood) and some of the variation. Perhaps we should have 4 wee columns to highlight this with an x if weighed to that measure Soosider3 (talk) 12:52, 15 December 2023 (UTC)

I don’t think 4 more columns is a good idea.

I’ve seen polls before in other contexts that randomise question orders for the reasons you mention. This is the same. I think we just have to take the poll at face value.

The links allow really keen readers to form their own opinions about polls.

If you want to add a brief note in the usual place highlighting what they have done I think that’s entirely reasonable. RERTwiki (talk) 10:13, 18 December 2023 (UTC)

Fully in agreement with RERTwiki here. Lets avoid any confirmation bias. AlloDoon (talk) 10:50, 20 December 2023 (UTC)

Oh behave yourself, lets report the data as it is presented in data tables and not as per client. Soosider3 (talk) 12:15, 20 December 2023 (UTC)

Aye, perhaps a note is the way to go at least we seem to have some consensus on that, I am aware of the asking the questions in a random order process, however that is not what happened here, where the same Q was asked at 2 different stages, in effect producing 2 small polls (at best) perhaps we should report them as reported by data tables ie as 2 polls rather than as client presented them, that is usually our default position - to go to the tables as most reliable source of data, information clearly presented in a manner that is obvious to the reader, rather than how the client would prefer it. Otherwise we are actually in danger of manipulating the data, check out the data tables pollster has presented them as seperate that is how we should record them

I see that Focaldata have at last published the tables on their website and it only took a complaint to BPC to get it done, perhaps not a ringing endorsement of them as political pollsters. Still no statement from them as to methodology used. Soosider3 (talk) 12:13, 20 December 2023 (UTC)

IPSOS poll removal

Happy New Year!!

I noticed when updating the charts that two IPSOS polls from back in '22 have been removed from the main table.

It doesn't make any difference to the charts, as they were already marked as non-standard questions and have never been included. I noticed they were missing when I didn't have to mark them for removal.

There is a comment in the edit history to the effect that they didn't ask about a future referendum, which seems quite unlikely, but might be grounds for excluding them.

I'm generally very negative on memory-holing interesting data, and would rather this was back as it was.

What do people think? RERTwiki (talk) 11:53, 17 January 2024 (UTC)

and a happy new year to you.

That's an odd one as notes field had already referred to it. Given length of time polls had been present I would assume our more regular editors were happy enough, therefore have reverted them. Soosider3 (talk) 12:18, 18 January 2024 (UTC)

'Additional Citations for Verification'

Eh? Anybody any idea what this is about?

Since essentially every poll is cited by a link to the originator, this makes little sense to me. RERTwiki (talk) 17:34, 30 March 2024 (UTC)

Absolutely agree with you, feels as if only the text has been read without reading and understanding the actual poll figures. Happy enough to remove as cant see way to actually engage otherwise. Soosider3 (talk) 13:18, 31 March 2024 (UTC)

Well, it's gone, I'm happy, thanks! RERTwiki (talk) 17:18, 9 April 2024 (UTC)

Having to suspend graphs

Hi - I’m temporally capacitated in producing the charts, potentially for a long time.

I think we need someone to work on the charts in similar vein. The current work to do by this largely on the unionist side makes a sensible approach when the majority of work on items is SnP.

Please reply if you are interested. RERTwiki (talk) 14:49, 19 April 2024 (UTC)

Hi RERTwiki, really sorry to hear that you are not able to continue to produce graphs, many thanks for all the time and effort you have given to this task, I for one really appreciate the effort taken.

Would be happy to take a look at what is involved before fully committing myself to task. Not sure how we would progress that but am open to suggestions. Soosider3 (talk) 10:15, 23 April 2024 (UTC)

Some how I ought to be able to supply all interested the source file of these graphs, which is an excel file, though someone needs to advise me how to. How do I get you (and others who require ) my email?

Still believe that people with different views maintaining data and graphs is a help. You getting the data in and me getting the graphs in is helpful, but que sera sera. RERTwiki (talk) 15:35, 25 April 2024 (UTC)

morning

had a look at how you might privately transfer data, looked at a few data sharing sites but they seem a bit of a palaver.

Simplest method might be to create a gmail account that you can delete once exercise completed

I have created one to receive files its indypolling@gmail.com, please feel free to send files I will acknowledge when recieved Soosider3 (talk) 07:24, 26 April 2024 (UTC)

I’ll send something asap, please can you make sure all

interested can seen the thing I send. RERTwiki (talk) 09:09, 26 April 2024 (UTC)

You can be absolutely sure that I will share file with any and all interested parties Soosider3 (talk) 15:36, 26 April 2024 (UTC)

Hello all, I have received the files from RERTwiki, bit of a saga as they went directly to spam and I've only just noticed them.

As promised I am happy to share the spreadsheet with any and all interested parties please just drop me a note at the above email indypolling@gmail.com

In the meantime a quick glance at the spreadsheet is testimony to just how much work RERTwiki has done on these over the years and we all owe him deep gratitude for his efforts.

I will explore the spreadsheet over the next few days and have a go at producing the graphs as per RERTwiki, it may be a good time once we are more familiar with the spreadsheet to have a discussion on how we see it developing, so any ideas would be warmly welcome. Soosider3 (talk) 11:44, 14 May 2024 (UTC)