Graduate Seminar: Compact Objects in Dense Stellar Systems
Omega Centauri Society Problem Sets
Problem Set 3 — Bayesian Null-Result Analysis
Estimated time: 3–4 hours Total marks: 40 Tools: workflow-null-result omegacentauri.me/pset-3-null-result.html
Learning objectives
  1. Derive single-transmitter detection probability from the Civiletti (2025) geometric sky fraction model
  2. Compute the Bayes factor K and posterior update from a null SETI result
  3. Convert a detection probability to a null-result significance in Gaussian σ units using the probit function
  4. Compare the inferential strength of different survey architectures (VLA/COSMIC vs. SKA projection)

Background

A null result in a SETI survey is not "no result" — it is evidence. The strength of that evidence depends on how likely we would have been to detect a signal if one existed. Civiletti (2025) provides a geometric model for the single-transmitter detection probability Pdetect that accounts for the fraction of sky covered, the fraction of frequency space searched, and the maximum geometric depth of the survey.

Single-transmitter detection probability (Civiletti 2025): P_single = f_sky · f_freq · min(1, 0.6 · δ / R_MW) where: f_sky = fraction of sky covered by the survey beam f_freq = fraction of monitored frequency space δ = survey depth (maximum distance at which a signal at the assumed EIRP would be detectable), ly R_MW = 52,000 ly (Milky Way half-light radius) For N independent transmitters, the probability of ≥ 1 detection: P_detect = 1 − (1 − P_single)^N Bayesian update from a non-detection: K = 1 − P_detect (Bayes factor for H₀: no signal) P_post = K · P₀ / (K · P₀ + (1 − P₀)) where P₀ is the prior probability that a transmitter exists. Null-result significance in Gaussian σ: σ_null = Φ⁻¹(P_detect) (probit of P_detect)

Problems

1. VLA/COSMIC null result: detection probability

The VLA/COSMIC 2025 survey (VLA Sky Survey with the Commensal Open-Source Multimode Interferometer Cluster) monitored Omega Centauri with the following parameters:

  • fsky = 0.75 (fraction of OC's angular area within primary beams)
  • ffreq = 0.10 (L and S band combined, relative to full radio spectrum)
  • δ = 1,000 ly (sensitivity limit for EIRP = 1014 W at 17,900 ly distance)
  • N = 10 (number of assumed transmitters in OC)
  1. Compute Psingle, Pdetect, and σnull by hand for these parameters. Show your work. [8 marks]
  2. Now use the OCS Null-Result workflow (VLA/COSMIC preset) to verify your answer. Do the numbers agree? If they differ by more than 5%, explain why. [3 marks]
  3. Is a σnull < 1 result strong evidence against the existence of an EIRP = 1014 W transmitter in OC? Why or why not? [4 marks]
↗ Null-Result Workflow — VLA/COSMIC preset (N=10, δ=1000 ly, f_sky=0.75, f_freq=0.10)
Show solution
Part (a): P_single = f_sky · f_freq · min(1, 0.6 · δ / R_MW) = 0.75 × 0.10 × min(1, 0.6 × 1000 / 52000) = 0.075 × min(1, 0.01154) = 0.075 × 0.01154 = 8.65 × 10⁻⁴ P_detect = 1 − (1 − 8.65×10⁻⁴)¹⁰ ≈ 1 − (0.999135)¹⁰ ≈ 1 − 0.99135 (using (1−x)^n ≈ 1−nx for small x) = 0.00865 or ~0.87% σ_null = Φ⁻¹(0.00865) ≈ −2.38σ Wait — the convention in the OCS tool is σ_null = Φ⁻¹(P_detect) measuring the significance of the expected detection as a positive number. For P_detect = 0.87%, Φ⁻¹(0.0087) ≈ −2.38, meaning the survey is ~2.4σ below the threshold where we'd expect to see a signal. This is a weak null result — if there's a transmitter, we'd have a <1% chance of seeing it anyway.
Part (b): The workflow should agree to within rounding. Any discrepancy >5% likely reflects a different convention for the probit function or a rounding of logN = log₁₀(10) = 1.0 exactly.

Part (c): No. A P_detect of <1% means that even if a 10¹⁴ W transmitter exists in OC, the VLA/COSMIC survey had only a 0.87% chance of detecting it. Not detecting it therefore provides very little Bayesian evidence against the transmitter hypothesis. The posterior probability barely changes from the prior. Only surveys that would have high P_detect (say >50%) if a transmitter existed can strongly constrain the hypothesis.
2. SKA projection and observational design

The Square Kilometre Array (SKA1-Mid) is projected to reach EIRP = 1013 W sensitivity at 17,900 ly with the following parameters:

  • fsky = 0.95 (near-complete sky coverage with commensal mode)
  • ffreq = 0.40 (broad 350 MHz – 14 GHz coverage)
  • δ = 10,000 ly (improved EIRP sensitivity)
  • N = 100 (log₁₀ N = 2; optimistic Drake estimate)
  1. Compute Pdetect for the SKA projection. How many σ does this correspond to? [6 marks]
  2. Compare the Bayes factor K for the SKA projection vs. VLA/COSMIC. By what factor does the SKA non-detection more strongly update the prior? [4 marks]
  3. What survey ffreq would be needed to achieve σnull ≥ 2.0, holding all other SKA parameters fixed? Solve analytically. [5 marks]
↗ Null-Result Workflow — SKA projection preset (N=100, δ=10,000 ly)
Show solution
Part (a): P_single = 0.95 × 0.40 × min(1, 0.6 × 10000 / 52000) = 0.38 × min(1, 0.1154) = 0.38 × 0.1154 = 0.04385 P_detect = 1 − (1 − 0.04385)¹⁰⁰ = 1 − (0.95615)¹⁰⁰ (0.95615)¹⁰⁰ = exp(100 × ln(0.95615)) = exp(100 × (−0.04482)) = exp(−4.482) = 0.01127 P_detect = 1 − 0.01127 = 0.9887 σ_null = Φ⁻¹(0.9887) ≈ 2.28σ The SKA null result would correspond to ~2.3σ significance.

Part (b): K_SKA = 1 − 0.9887 = 0.0113; K_VLA ≈ 1 − 0.0087 ≈ 0.9913. The Bayes factor ratio is K_SKA/K_VLA = 0.0113/0.9913 ≈ 0.0114. The SKA non-detection is ~88× more powerful evidence against the transmitter hypothesis (posterior updates 88× more strongly per unit prior).

Part (c): We want P_detect = Φ(2.0) = 0.9772, so (1 − P_single)¹⁰⁰ = 0.0228. Taking ln: 100 × ln(1 − P_single) = ln(0.0228) = −3.78. So ln(1 − P_single) = −0.0378, giving 1 − P_single = 0.9629, P_single = 0.0371. Then f_freq = P_single / (f_sky × min(1, 0.6 δ/R_MW)) = 0.0371 / (0.95 × 0.1154) = 0.0371 / 0.1096 ≈ 0.338 — so f_freq ≥ 0.34 (34% frequency coverage) is needed.
3. Bayesian prior sensitivity

The posterior update depends critically on the prior P₀. Use the SKA projection (P_detect ≈ 0.989) and compute the posterior Ppost for three priors, by hand and then verify with the tool.

  1. Compute Ppost for P₀ = 0.5 (agnostic), P₀ = 0.1 (skeptical), and P₀ = 0.01 (very skeptical). Show how dramatically the posterior changes with the prior. [5 marks]
  2. For P₀ = 0.01, a skeptic notes: "Even after the SKA null result, the posterior is still close to the prior — it barely changes." Explain mathematically why this happens when P₀ is very small. [5 marks]
Show solution
Part (a): K = 1 − P_detect = 0.0113. P_post = K · P₀ / (K · P₀ + (1 − P₀)) P₀ = 0.5: P_post = 0.0113 × 0.5 / (0.0113 × 0.5 + 0.5) = 0.00565 / (0.00565 + 0.5) = 0.00565 / 0.5057 ≈ 0.0112 → dropped from 50% to 1.1% P₀ = 0.1: P_post = 0.0113 × 0.1 / (0.0113 × 0.1 + 0.9) = 0.00113 / (0.00113 + 0.9) = 0.00113 / 0.9011 ≈ 0.00125 → dropped from 10% to 0.125% P₀ = 0.01: P_post = 0.0113 × 0.01 / (0.0113 × 0.01 + 0.99) = 1.13×10⁻⁴ / (1.13×10⁻⁴ + 0.99) = 1.13×10⁻⁴ / 0.99011 ≈ 1.14×10⁻⁴ → barely changed from 1%
Part (b): When P₀ ≪ 1, the denominator is dominated by (1 − P₀) ≈ 1. So P_post ≈ K · P₀ / 1 = K · P₀. This means the posterior is simply K times the prior. With K = 0.011 and P₀ = 0.01, P_post ≈ 0.011 × 0.01 = 1.1 × 10⁻⁴ — a factor of ~90 reduction. But since the absolute value is so small (0.01% → 0.01%), a skeptic measuring percentage change would say "it went from 1% to 0.01%, which is a tiny absolute change." The real lesson is that the ratio P_post/P₀ = K does not depend on P₀; what changes is the absolute posterior, which is tiny when the prior is tiny. A Bayesian analysis cannot make a hypothesis plausible if the prior is very small — it can only confirm or refute existing probability.

References