Real networks fail in repetitive ways. This module gives you the symptom → check → likely cause method that support engineers use, plus the vocabulary of the jobs in this industry.
1. The method
- Define the symptom exactly — who (one user, one cell, one region, everyone), what (calls, SMS, data), since when, how often.
- Locate the layer — radio, transport, core signalling, user plane, application/IMS, billing/policy, device/SIM.
- Compare with a working case — same procedure in another cell or subscriber.
- Look for the first error in traces/logs (not the loudest).
- Change one thing at a time and re-test; document.
- Escalate with evidence — IMSI/MSISDN, time (with time zone), cell ID, trace file, error codes, screenshots.
OSI-ish telecom stack for debugging: Device & SIM → Radio (RSRP/SINR) → Access signalling (RRC/NAS) → Transport (IP/MPLS/fibre) → Core control (MME/AMF, HSS/UDM) → User plane (SGW/PGW/UPF) → Services (IMS, SMSC, DNS) → Policy/charging.
2. Symptom table
| Symptom | First checks | Likely causes |
|---|---|---|
| No service / "Emergency calls only" | SIM inserted? Roaming allowed? Bands supported? Attach reject cause | Barred SIM, IMEI blacklisted, no coverage, PLMN not allowed, HLR/HSS entry missing |
| Attach rejected, EMM #7 / #8 | HSS profile: EPS subscription, APN | LTE not provisioned on subscription |
| Attach rejected, #3 / #6 | IMSI/IMEI status in HSS/EIR | Illegal UE/ME |
| Attach OK but no internet | APN, PDP/PDU session, DNS, IP address, QoS | Wrong APN (ESM #27), session rejected, DNS down, firewall/NAT, UPF/PGW issue, policy rule blocking, exhausted CGNAT ports |
| Slow data | RSRP/SINR, PRB load, backhaul, TCP, throttling policy | Congested cell, cell edge, QoS (AMBR) cap, transport congestion, inadequate MTU, peering |
| Data works, specific site not | DNS, MTU/MSS, IPv6/NAT64, CGNAT | Path MTU, DNS64 mis-config, blocked IPs |
| Latency high / jitter | Ping to first hop and beyond, radio retransmissions | Cell load, backhaul bottleneck, routing detour, bufferbloat |
| SMS not sent (MO) | Phone's SMSC number, SMPP/MAP traces, SMSC queues | Wrong SMSC address, barred, prepaid balance, SMSC down |
| SMS not delivered (MT) | SRI-SM result, absent subscriber, home routing, firewall | Phone off (6), unknown subscriber (1), SMSC rejects, SMS firewall block (REJECTD), full memory |
| OTP late/missing | A2P route, DLR statuses, hop count, sender ID | Grey route, throttling, filtering, roaming user not reachable |
| Call fails immediately | Cause code (Q.850, SIP) | 17 busy, 18 no answer, 34 congestion, 38 out of order, 404/480/503 |
| One-way audio | RTP path both directions, NAT, SDP IP, firewall | SBC/NAT misconfiguration, blocked RTP ports, wrong codec |
| No audio at all | SDP negotiation, dedicated bearer set up? | Dedicated bearer failure (QCI 1), media gateway down, SRTP mismatch |
| Call drops at ~30 s / ~3 min | SIP timers, ACK/BYE, session refresh | NAT keepalive, firewall closing UDP, session-timer mismatch |
| VoLTE not available | IMS registration, ims APN, device profile |
VoLTE flag not provisioned, IMS core down, roaming without VoLTE agreement |
| Poor voice quality | RTCP stats, MOS, packet loss/jitter, codec | Radio loss, QoS not applied, transcoding |
| Roaming fails | Roaming agreement (IR.21), HLR/HSS status, SS7/Diameter route, SIM profile | PLMN not allowed, barring, GT/realm routing, missing TADIG/IMSI ranges, firewall blocking |
| Roamer cannot call home | CAMEL/CAP, VPLMN, IMS roaming (S8HR) | CAMEL triggers missing, HR/LBO issue |
| Number portability failure | NP database, routing | Ported number not updated |
| Charging issues (balance, bills) | OCS logs, CDRs, Gy/Ro traces | Session not closed, rating plan wrong, CDR gaps |
| Everything down at one site | Alarms, power, transport, sync | Power cut, fibre cut, transport link, GPS lost, hardware |
| Many sites down in an area | Aggregation link, controller/core node | Fibre cut, router failure, core node outage |
| Core signalling overload | Attach storm, retries, SCTP drops | Mass reboot, firmware bug, DDoS/botnet, misbehaving app, missing back-off |
| Sudden spike in SRI/ATI | Source GT/realm, counters | Tracking attempt or scanning, aggregator probing |
| Many failed OTP / high SMS cost | Destination countries, sender | SMS pumping/AIT |
| Subscriber says "someone hacked my SMS" | Location updates from foreign GT, SIM-swap logs, call forwarding | SS7 interception, SIM swap, account takeover, malware |
3. Alarms and health
| Term | Meaning |
|---|---|
| Alarm severity | Critical, major, minor, warning, cleared |
| Heartbeat / keepalive | SCTP, BFD, Diameter DWR/DWA (Device-Watchdog), GTP echo |
| Link state | Up/down/flapping |
| Sync loss | Cell goes off air when GPS/PTP clock lost |
| VSWR alarm | Antenna/feeder problem |
| Overload / congestion alarm | CPU, memory, signalling rate, bearers |
| Capacity thresholds | PRB utilisation > 70–80% triggers expansion |
| Counter, KPI, trace | Aggregated statistics vs per-subscriber trace |
| Trace types | Subscriber trace (IMSI), cell trace, call trace (e.g. 3GPP Trace feature, MDT minimization of drive tests) |
| NOC procedures | Incident, change, problem management (ITIL); RCA (Root-Cause Analysis); post-mortem |
| MTTR / MTBF / SLA | Mean Time To Repair / Between Failures / Service Level Agreement |
| Change window | Planned maintenance at night with rollback plan |
| Disaster recovery | RTO (recovery time objective) / RPO (recovery point objective) |
4. Typical job roles
| Role | What they do |
|---|---|
| RF / radio planning engineer | Coverage, capacity, parameters, drive tests |
| Core network engineer | MME/AMF/SMF/UPF, HSS/UDM, IMS, SS7/Diameter routing |
| Transport / IP engineer | Fibre, microwave, MPLS, BGP, timing |
| Voice / IMS engineer | SIP, SBC, VoLTE, interconnect |
| Signalling / interconnect engineer | SS7, Diameter, SIGTRAN, roaming |
| Messaging engineer | SMSC, SMPP, A2P, MMS, RCS |
| Security engineer / SOC analyst | Firewalls, fraud, threat intelligence, incident response |
| Revenue assurance / fraud analyst | Detect revenue leakage and fraud |
| Billing & OSS/BSS engineer | Rating, charging, provisioning |
| Cloud / NFV / DevOps engineer | Kubernetes, CI/CD, automation, observability |
| Network operations (NOC) engineer | 24/7 monitoring, incidents |
| Field engineer / site technician | Tower, installation, fibre splicing |
| Product / solution architect | Designs services and architectures |
| Regulatory & compliance officer | Licences, LI, GDPR |
| Wholesale / roaming manager | Agreements, IR.21, IOT tariffs |
| Pre-sales / vendor engineer | Nokia, Ericsson, Huawei, ZTE, Samsung, Ciena, Cisco, Juniper, Mavenir, Metaswitch, Oracle, Mobileum, Enea, Infobip |
| Standards / research | 3GPP delegates, universities |
5. Learning path and certifications
| Track | Examples |
|---|---|
| Foundations | This guide; GSMA training; ETSI/3GPP intro courses |
| Vendor certs | Nokia (NRS), Ericsson (ECS), Huawei (HCIA/HCIP-5G), Cisco (CCNA/CCNP SP) |
| Radio | RF planning courses, TEMS/Nemo, Atoll/Planet |
| Networking | CCNA, JNCIA, MEF-CECP, RHCSA |
| Cloud/telco | CKA (Kubernetes), OpenStack, ONAP |
| Security | GSMA FS documents, GIAC, CISSP, 5G security courses |
| Open source labs | Open5GS, free5GC, srsRAN, Osmocom |
| Reading | 3GPP specs by Release, GSMA PRDs, IETF RFCs, vendor documentation |
6. Mental models to remember
- Find → authenticate → authorise → route → charge. Every service follows this chain.
- Control plane vs user plane. Signalling decides, user plane carries.
- Home vs visited. Almost every roaming problem is a home/visited disagreement.
- Radio is lossy, core is deterministic. If it is intermittent, suspect radio or transport first.
- Timers tell the story. Gaps of 3 s, 5 s, 10 s, 30 s often match known protocol timers.
- Trust nothing at the border. Anything coming from outside must be validated.
- Capacity problems look like random failures. Look at load counters before changing configuration.
- Backward compatibility is everywhere. Old SS7, 2G devices and phones with odd firmware will surprise you.
Remember this: troubleshooting is not magic; it is a method (symptom → layer → trace → first error → fix → verify) plus a vocabulary of causes. You now have both.