[PATCH 00/32] s390: Improve exception table handling and disassembler

[PATCH 00/32] s390: Improve exception table handling and disassembler

[Heiko Carstens]

Extend the existing get_trap_ip() so that it returns the address of
the instruction which caused an exception, and use that for both the
disassembler as well as exception handling.

This allows improving two things:

- The in kernel disassembler prints PGM+PSW markers which clearly
  identify the instruction causing an exception, and the current PSW
  address. This replaces the previous "best guess" approach with '*' and
  '>' markers

- EX_TABLE entries for inline assemblies are simplified. Only the address
  of the faulting instruction needs to be provided from now on. The second
  entry, which is the address following the faulting instruction, is not
  needed anymore

Disassembler
------------

The output of the kernel disassembler changes. If the disassembly is
printed as a result of a program check, 'PGM' and 'PSW' markers identify
the instruction which caused the exception, and where the PSW points
to. For suppressing exceptions, the output now looks like this:

 Krnl Code:
            000003ff8dc7bff8: e310f0080004 lg      %r1,8(%r15)
       PGM >000003ff8dc7bffe: e3a0e0000004 lg      %r10,0(%r14)
       PSW >000003ff8dc7c004: a7280000     lhi     %r2,0
            000003ff8dc7c008: eb9ff0900024 stmg    %r9,%r15,144(%r15)

For nullifying exceptions where both addresses are the same,
'PGM+PSW >' is printed instead:

 Krnl Code:
            000003ff8dc7bff8: e310f0080004 lg      %r1,8(%r15)
   PGM+PSW >000003ff8dc7bffe: e3a0e0000004 lg      %r10,0(%r14)
            000003ff8dc7c004: a7280000     lhi     %r2,0
            000003ff8dc7c008: eb9ff0900024 stmg    %r9,%r15,144(%r15)

EX_TABLE entries
----------------

With get_trap_ip() now returning the exact faulting instruction address,
EX_TABLE entries for inline assemblies are simplified. Only the address
of the faulting instruction needs to be provided. The second entry - the
address following the faulting instruction - is no longer needed.

For example, an inline assembly using a single instruction can now be
written as:

        asm_inline volatile(
                "0:     lg      %[data],%[addr]\n"
                "1:\n"
                EX_TABLE_INSN(0b, 1b));

Previously this required:

        asm_inline volatile(
                "       nopr    %%r7\n"
                "0:     lg      %[data],%[addr]\n"
                "1:     nopr    %%r7\n"
                EX_TABLE(0b, 1b)
                EX_TABLE(1b, 1b));

The old approach had several non-obvious restrictions:

- If an instruction immediately preceding the inline assembly caused a
  non-nullifying exception, the PSW would point to the start of the
  inline assembly. If the faulting instruction was placed there, the
  exception handler would incorrectly resolve the fault using the inline
  assembly's fixup handler.  Hence, a leading NOP was required before the
  faulting instruction

- Similarly, an exception table entry placed behind a faulting
  instruction must also point to an instruction that cannot fault. A
  nullifying exception on the following instruction would otherwise be
  incorrectly resolved by that entry

- Furthermore, instructions that can cause both nullifying and
  non-nullifying exceptions required two separate exception table entries

These restrictions are error-prone and have caused several bugs in the
past.

All EX_TABLE* macros have been renamed to EX_TABLE_INSN*. This allows the
old and new semantics to be distinguished, and ensures that backports of
code using the new semantics will intentionally fail to compile if the
new infrastructure is not present.

Note
----

There are some existing oddities and corner cases with kprobes. Only
some of them will be addressed with this series. The other ones will
be addressed with follow-on changes, since this series is already
large enough.

Thanks,
Heiko

Heiko Carstens (32):
  s390/kprobes: Prevent kprobes on instructions with exception table entry
  s390/early: Fix early warning handling
  s390: Clear pt_regs flags field
  s390/bug: Indicate modified PSW in disassembly
  s390/tools: Rename gen_facilities to gen_bitmaps
  s390/tools: Generate bitmap of nullified program checks
  s390/traps: Handle nullifying program checks in get_trap_ip()
  s390/dis: Improve show_code() output
  s390/extable: Rework exception handling logic
  s390/diag: Convert amode31 exception tables to EX_TABLE_INSN
  s390/uaccess: Convert inline assembly to EX_TABLE_INSN
  s390/lib/uaccess: Convert inline assembly to EX_TABLE_INSN
  s390/futex: Convert inline assembly to EX_TABLE_INSN
  s390/diag: Convert inline assembly to EX_TABLE_INSN
  s390/mm/pfault: Convert inline assembly to EX_TABLE_INSN
  s390/cpu_mf: Convert inline assembly to EX_TABLE_INSN
  s390/cert_store: Convert inline assembly to EX_TABLE_INSN
  s390/ipl: Convert inline assembly to EX_TABLE_INSN
  s390/hypfs: Convert inline assembly to EX_TABLE_INSN
  s390/ap: Convert inline assembly to EX_TABLE_INSN
  s390/fpu: Convert inline assembly to EX_TABLE_INSN
  s390/traps: Convert inline assembly to EX_TABLE_INSN
  s390/word-at-a-time: Convert inline assembly to EX_TABLE_INSN
  s390/kvm: Convert inline assembly to EX_TABLE_INSN
  s390/pci: Convert inline assembly to EX_TABLE_INSN
  s390/boot: Convert inline assembly to EX_TABLE_INSN
  s390/cio: Convert inline assembly to EX_TABLE_INSN
  s390/dasd: Convert inline assembly to EX_TABLE_INSN
  s390/diag_ftp: Convert inline assembly to EX_TABLE_INSN
  s390/sclp: Convert inline assembly to EX_TABLE_INSN
  s390/bpf: Simplify exception table entry handling
  s390/extable: Remove EX_TABLE macros and fallback extable search

 arch/s390/boot/ipl_parm.c                     |   6 +-
 arch/s390/boot/pgm_check.c                    |  14 ++-
 arch/s390/boot/physmem_info.c                 |  18 +--
 arch/s390/boot/startup.c                      |  12 +-
 arch/s390/hypfs/hypfs_vm.c                    |   6 +-
 arch/s390/include/asm/Kbuild                  |   1 +
 arch/s390/include/asm/ap.h                    |  12 +-
 arch/s390/include/asm/asm-extable.h           |  28 +++--
 arch/s390/include/asm/cpu_mf.h                |  18 +--
 arch/s390/include/asm/diag.h                  |   5 +-
 arch/s390/include/asm/diag288.h               |   6 +-
 arch/s390/include/asm/extable.h               |  25 ++---
 arch/s390/include/asm/fpu-insn.h              |   6 +-
 arch/s390/include/asm/futex.h                 |  17 ++-
 arch/s390/include/asm/ptrace.h                |   6 +
 arch/s390/include/asm/uaccess.h               |  45 +++-----
 arch/s390/include/asm/word-at-a-time.h        |   5 +-
 arch/s390/kernel/cert_store.c                 |   6 +-
 arch/s390/kernel/diag/diag.c                  |  12 +-
 arch/s390/kernel/dis.c                        |  35 +++---
 arch/s390/kernel/dumpstack.c                  |   7 +-
 arch/s390/kernel/early.c                      |  26 ++++-
 arch/s390/kernel/entry.S                      |   1 +
 arch/s390/kernel/ipl.c                        |   6 +-
 arch/s390/kernel/kprobes.c                    |   3 +
 arch/s390/kernel/text_amode31.S               |  34 +++---
 arch/s390/kernel/traps.c                      |  30 +++--
 arch/s390/kvm/s390/gaccess.c                  |  11 +-
 arch/s390/lib/uaccess.c                       |  29 ++---
 arch/s390/mm/extable.c                        |   2 +-
 arch/s390/mm/pfault.c                         |  12 +-
 arch/s390/net/bpf_jit_comp.c                  |  78 ++++---------
 arch/s390/pci/pci_clp.c                       |  12 +-
 arch/s390/pci/pci_insn.c                      |  36 +++---
 arch/s390/pci/pci_mmio.c                      |  28 ++---
 arch/s390/tools/.gitignore                    |   1 +
 arch/s390/tools/Makefile                      |  16 ++-
 .../tools/{gen_facilities.c => gen_bitmaps.c} | 103 ++++++++++++++----
 drivers/s390/block/dasd_diag.c                |   6 +-
 drivers/s390/char/diag_ftp.c                  |   6 +-
 drivers/s390/char/sclp.h                      |   7 +-
 drivers/s390/cio/ioasm.c                      |  24 ++--
 42 files changed, 409 insertions(+), 352 deletions(-)
 rename arch/s390/tools/{gen_facilities.c => gen_bitmaps.c} (65%)

-- 
2.53.0

[PATCH 01/32] s390/kprobes: Prevent kprobes on instructions with exception table entry

[Heiko Carstens]

The mvcos exception handler ex_handler_ua_mvcos() decodes the faulting
instruction, assuming it is an mvcos instruction. In case the instruction
is kprobed the decoded instruction is a breakpoint instruction instead,
which leads to incorrect instruction decoding and potential register
corruption.

Fix this by simply preventing to set a kprobe on such instructions,
similar like arm64 is doing it.

Fixes: c488f5187a24 ("s390/uaccess: Shorten raw_copy_from_user() / raw_copy_to_user() inline assemblies")
Cc: stable@vger.kernel.org
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/kprobes.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/s390/kernel/kprobes.c b/arch/s390/kernel/kprobes.c
index c450120b4474..e806b124e4ba 100644
--- a/arch/s390/kernel/kprobes.c
+++ b/arch/s390/kernel/kprobes.c
@@ -85,6 +85,9 @@ static bool can_probe(unsigned long paddr)
 	if (!kallsyms_lookup_size_offset(paddr, NULL, &offset))
 		return false;
 
+	if (s390_search_extables(paddr))
+		return false;
+
 	/* Decode instructions */
 	addr = paddr - offset;
 	while (addr < paddr) {
-- 
2.53.0

[PATCH 02/32] s390/early: Fix early warning handling

[Heiko Carstens]

When adding support for __WARN_printf() the early program check handler was
not taken into account. Add the missing support to the early program check
handler. This avoids that the kernel crashes instead of printing a warning.

Fixes: 04dabb4261c3 ("s390/bug: Implement __WARN_printf()")
Fixes: 79996065cfa2 ("s390/bug: Skip __WARN_trap() in call traces")
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/early.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c
index b27239c03d79..1fc153d8385b 100644
--- a/arch/s390/kernel/early.c
+++ b/arch/s390/kernel/early.c
@@ -153,16 +153,25 @@ static __init void setup_topology(void)
 void __init __do_early_pgm_check(struct pt_regs *regs)
 {
 	struct lowcore *lc = get_lowcore();
-	unsigned long ip;
 
 	regs->int_code = lc->pgm_int_code;
 	regs->int_parm_long = lc->trans_exc_code;
 	regs->last_break = lc->pgm_last_break;
-	ip = __rewind_psw(regs->psw, regs->int_code >> 16);
+	regs->monitor_code = lc->monitor_code;
 
 	/* Monitor Event? Might be a warning */
 	if ((regs->int_code & PGM_INT_CODE_MASK) == 0x40) {
-		if (report_bug(ip, regs) == BUG_TRAP_TYPE_WARN)
+		enum bug_trap_type btt;
+		unsigned long ip;
+
+		if (regs->monitor_code == MONCODE_BUG_ARG) {
+			regs->psw.addr = regs->gprs[14];
+			btt = report_bug_entry((struct bug_entry *)regs->gprs[2], regs);
+		} else {
+			ip = __rewind_psw(regs->psw, regs->int_code >> 16);
+			btt = report_bug(ip, regs);
+		}
+		if (btt == BUG_TRAP_TYPE_WARN)
 			return;
 	}
 	if (fixup_exception(regs))
-- 
2.53.0

[PATCH 03/32] s390: Clear pt_regs flags field

[Heiko Carstens]

Clear the pt_regs flags field in the early program check handler as well
as in the stack_invalid handler. Without this the pt_regs flags field will
have random values, depending on previous stack contents.

This matters currently for bug reporting and disassembly, where flags like
PIF_PSW_ADDR_ADJUSTED may then be incorrectly set, which could lead to a
partially incorrect disassembly or wrong code markers.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/early.c | 1 +
 arch/s390/kernel/entry.S | 1 +
 2 files changed, 2 insertions(+)

diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c
index 1fc153d8385b..b8e3c065fbec 100644
--- a/arch/s390/kernel/early.c
+++ b/arch/s390/kernel/early.c
@@ -154,6 +154,7 @@ void __init __do_early_pgm_check(struct pt_regs *regs)
 {
 	struct lowcore *lc = get_lowcore();
 
+	regs->flags = 0;
 	regs->int_code = lc->pgm_int_code;
 	regs->int_parm_long = lc->trans_exc_code;
 	regs->last_break = lc->pgm_last_break;
diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
index 10dd9bbdf985..f51ecd8e7d21 100644
--- a/arch/s390/kernel/entry.S
+++ b/arch/s390/kernel/entry.S
@@ -621,6 +621,7 @@ SYM_CODE_START(stack_invalid)
 	stmg	%r0,%r7,__PT_R0(%r11)
 	stmg	%r8,%r9,__PT_PSW(%r11)
 	mvc	__PT_R8(64,%r11),0(%r14)
+	xc	__PT_FLAGS(8,%r11),__PT_FLAGS(%r11)
 	GET_LC	%r2
 	mvc	__PT_ORIG_GPR2(8,%r11),__LC_PGM_LAST_BREAK(%r2)
 	xc	__SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
-- 
2.53.0

[PATCH 04/32] s390/bug: Indicate modified PSW in disassembly

[Heiko Carstens]

With [1] the PSW address is set to the return address of the calling
__WARN_trap() function in order to provide a more useful disassembly of
the caller rather than the helper itself.

However, since regs->psw.addr is modified, regs->int_code has nothing to
do with the exception. In order to make that more obvious change the
disassembler output.

Introduce a new PIF_WARN_TRAP_SKIPPED pt_regs flag and set it whenever
__WARN_trap() is skipped. In the disassembler output print a "WARN RET"
prefix to make it obvious that the PSW points to the return address of
the warning trap function.

Also do not print the interruption code and instruction length code for
this case, since they have nothing to do with the modified PSW.

Example output:

Krnl Code:
           000003ffe185bc1c: e3f0ff98ff71 lay     %r15,-104(%r15)
           000003ffe185bc22: e3e0f0980024 stg     %r14,152(%r15)
           000003ffe185bc28: c0e5ff45ed94 brasl   %r14,000003ffe0119750
 WARN RET >000003ffe185bc2e: c0e5ffa052b9 brasl   %r14,000003ffe0c661a0
           000003ffe185bc34: c020fffe86d6 larl    %r2,000003ffe182c9e0
           000003ffe185bc3a: e548f0a80006 mvghi   168(%r15),6
           000003ffe185bc40: e548f0a00005 mvghi   160(%r15),5
           000003ffe185bc46: a7690004     lghi    %r6,4

[1] 79996065cfa2 ("s390/bug: Skip __WARN_trap() in call traces")

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/ptrace.h |  2 ++
 arch/s390/kernel/dis.c         | 23 ++++++++++-------------
 arch/s390/kernel/dumpstack.c   |  7 +++++--
 arch/s390/kernel/early.c       |  9 +++++++--
 arch/s390/kernel/traps.c       |  1 +
 5 files changed, 25 insertions(+), 17 deletions(-)

diff --git a/arch/s390/include/asm/ptrace.h b/arch/s390/include/asm/ptrace.h
index 6411e3584283..c844b8aa0fd9 100644
--- a/arch/s390/include/asm/ptrace.h
+++ b/arch/s390/include/asm/ptrace.h
@@ -17,11 +17,13 @@
 #define PIF_PSW_ADDR_ADJUSTED		1	/* psw address has been adjusted */
 #define PIF_SYSCALL_RET_SET		2	/* return value was set via ptrace */
 #define PIF_FTRACE_FULL_REGS		3	/* all register contents valid (ftrace) */
+#define PIF_WARN_TRAP_SKIPPED		4	/* __WARN_trap() was skipped */
 
 #define _PIF_SYSCALL			BIT(PIF_SYSCALL)
 #define _PIF_ADDR_PSW_ADJUSTED		BIT(PIF_PSW_ADDR_ADJUSTED)
 #define _PIF_SYSCALL_RET_SET		BIT(PIF_SYSCALL_RET_SET)
 #define _PIF_FTRACE_FULL_REGS		BIT(PIF_FTRACE_FULL_REGS)
+#define _PIF_WARN_TRAP_SKIPPED		BIT(PIF_WARN_TRAP_SKIPPED)
 
 #define PSW32_MASK_PER		_AC(0x40000000, UL)
 #define PSW32_MASK_DAT		_AC(0x04000000, UL)
diff --git a/arch/s390/kernel/dis.c b/arch/s390/kernel/dis.c
index 1cec93895b3a..5a1a24d211c3 100644
--- a/arch/s390/kernel/dis.c
+++ b/arch/s390/kernel/dis.c
@@ -540,34 +540,31 @@ void show_code(struct pt_regs *regs)
 		start += 2;
 	}
 	/* Decode the instructions. */
-	ptr = buffer;
-	ptr += sprintf(ptr, "%s Code:", mode);
+	printk("%s Code:\n", mode);
 	hops = 0;
 	while (start < end && hops < 8) {
+		ptr = buffer;
 		opsize = insn_length(code[start]);
-		if  (start + opsize == 32)
-			*ptr++ = '*';
+		if (start == 32 && test_pt_regs_flag(regs, PIF_WARN_TRAP_SKIPPED))
+			ptr += sprintf(ptr, " WARN RET >");
+		else if (start + opsize == 32 && !test_pt_regs_flag(regs, PIF_WARN_TRAP_SKIPPED))
+			ptr += sprintf(ptr, "          *");
 		else if (start == 32)
-			*ptr++ = '>';
+			ptr += sprintf(ptr, "          >");
 		else
-			*ptr++ = ' ';
+			ptr += sprintf(ptr, "           ");
 		addr = pswaddr + start - 32;
 		ptr += sprintf(ptr, "%px: ", (void *)addr);
 		if (start + opsize >= end)
 			break;
 		for (i = 0; i < opsize; i++)
 			ptr += sprintf(ptr, "%02x", code[start + i]);
-		*ptr++ = '\t';
-		if (i < 6)
-			*ptr++ = '\t';
+		ptr += sprintf(ptr, "%*s", 13 - 2 * opsize, " ");
 		ptr += print_insn(ptr, code + start, addr);
 		start += opsize;
-		pr_cont("%s", buffer);
-		ptr = buffer;
-		ptr += sprintf(ptr, "\n          ");
+		pr_cont("%s\n", buffer);
 		hops++;
 	}
-	pr_cont("\n");
 }
 
 void print_fn_code(unsigned char *code, unsigned long len)
diff --git a/arch/s390/kernel/dumpstack.c b/arch/s390/kernel/dumpstack.c
index f9d52e05e01e..4ec08264b3ef 100644
--- a/arch/s390/kernel/dumpstack.c
+++ b/arch/s390/kernel/dumpstack.c
@@ -203,8 +203,11 @@ void __noreturn die(struct pt_regs *regs, const char *str)
 	console_verbose();
 	spin_lock_irq(&die_lock);
 	bust_spinlocks(1);
-	printk("%s: %04x ilc:%d [#%d]", str, regs->int_code & 0xffff,
-	       regs->int_code >> 17, ++die_counter);
+	if (test_pt_regs_flag(regs, PIF_WARN_TRAP_SKIPPED))
+		printk("%s: [#%d]", str, ++die_counter);
+	else
+		printk("%s: %04x ilc:%d [#%d]", str, regs->int_code & 0xffff,
+		       regs->int_code >> 17, ++die_counter);
 	pr_cont("SMP ");
 	if (debug_pagealloc_enabled())
 		pr_cont("DEBUG_PAGEALLOC");
diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c
index b8e3c065fbec..d875b3b304be 100644
--- a/arch/s390/kernel/early.c
+++ b/arch/s390/kernel/early.c
@@ -167,6 +167,7 @@ void __init __do_early_pgm_check(struct pt_regs *regs)
 
 		if (regs->monitor_code == MONCODE_BUG_ARG) {
 			regs->psw.addr = regs->gprs[14];
+			set_pt_regs_flag(regs, PIF_WARN_TRAP_SKIPPED);
 			btt = report_bug_entry((struct bug_entry *)regs->gprs[2], regs);
 		} else {
 			ip = __rewind_psw(regs->psw, regs->int_code >> 16);
@@ -183,8 +184,12 @@ void __init __do_early_pgm_check(struct pt_regs *regs)
 	 * some basic information in case it is too early for printk().
 	 */
 	register_early_console();
-	early_printk("PANIC: early exception %04x PSW: %016lx %016lx\n",
-		     regs->int_code & 0xffff, regs->psw.mask, regs->psw.addr);
+	if (test_pt_regs_flag(regs, PIF_WARN_TRAP_SKIPPED))
+		early_printk("PANIC: early exception PSW: %016lx %016lx\n",
+			     regs->psw.mask, regs->psw.addr);
+	else
+		early_printk("PANIC: early exception %04x PSW: %016lx %016lx\n",
+			     regs->int_code & 0xffff, regs->psw.mask, regs->psw.addr);
 	show_regs(regs);
 	disabled_wait();
 }
diff --git a/arch/s390/kernel/traps.c b/arch/s390/kernel/traps.c
index b6ba4465f59d..bfcab4592299 100644
--- a/arch/s390/kernel/traps.c
+++ b/arch/s390/kernel/traps.c
@@ -267,6 +267,7 @@ static void monitor_event_exception(struct pt_regs *regs)
 		return;
 	if (regs->monitor_code == MONCODE_BUG_ARG) {
 		regs->psw.addr = regs->gprs[14];
+		set_pt_regs_flag(regs, PIF_WARN_TRAP_SKIPPED);
 		btt = report_bug_entry((struct bug_entry *)regs->gprs[2], regs);
 	} else {
 		btt = report_bug(regs->psw.addr - (regs->int_code >> 16), regs);
-- 
2.53.0

[PATCH 05/32] s390/tools: Rename gen_facilities to gen_bitmaps

[Heiko Carstens]

Rename gen_facilities.c to gen_bitmaps.c and make it more generic so it
can be used for various bitmaps. Add command line parsing, and use it
to explicitly specify that the facilities header should be generated.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/tools/.gitignore                    |  1 +
 arch/s390/tools/Makefile                      |  8 +--
 .../tools/{gen_facilities.c => gen_bitmaps.c} | 49 +++++++++++++------
 3 files changed, 38 insertions(+), 20 deletions(-)
 rename arch/s390/tools/{gen_facilities.c => gen_bitmaps.c} (83%)

diff --git a/arch/s390/tools/.gitignore b/arch/s390/tools/.gitignore
index e6af51d9d183..e07d27143172 100644
--- a/arch/s390/tools/.gitignore
+++ b/arch/s390/tools/.gitignore
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0-only
 gen_facilities
+gen_bitmaps
 gen_opcode_table
 relocs
diff --git a/arch/s390/tools/Makefile b/arch/s390/tools/Makefile
index f2862364fb42..ac267bb46550 100644
--- a/arch/s390/tools/Makefile
+++ b/arch/s390/tools/Makefile
@@ -10,17 +10,17 @@ PHONY += kapi
 
 kapi:	$(kapi-hdrs-y)
 
-hostprogs		    += gen_facilities
+hostprogs		    += gen_bitmaps
 hostprogs		    += gen_opcode_table
 
-HOSTCFLAGS_gen_facilities.o += $(LINUXINCLUDE)
+HOSTCFLAGS_gen_bitmaps.o += $(LINUXINCLUDE)
 
-filechk_facility-defs.h = $(obj)/gen_facilities
+filechk_facility-defs.h = $(obj)/gen_bitmaps --facility
 
 filechk_dis-defs.h = \
 	$(obj)/gen_opcode_table < $(srctree)/arch/$(ARCH)/tools/opcodes.txt
 
-$(kapi)/facility-defs.h: $(obj)/gen_facilities FORCE
+$(kapi)/facility-defs.h: $(obj)/gen_bitmaps FORCE
 	$(call filechk,facility-defs.h)
 
 $(kapi)/dis-defs.h: $(obj)/gen_opcode_table FORCE
diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_bitmaps.c
similarity index 83%
rename from arch/s390/tools/gen_facilities.c
rename to arch/s390/tools/gen_bitmaps.c
index 32dd5a57240d..4bc5f19dd405 100644
--- a/arch/s390/tools/gen_facilities.c
+++ b/arch/s390/tools/gen_bitmaps.c
@@ -1,24 +1,25 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * Simple program to generate defines out of facility lists that use the bit
- * numbering scheme from the Princples of Operations: most significant bit
- * has bit number 0.
+ * Simple program to generate bitmaps, for example facility lists.
  *
  *    Copyright IBM Corp. 2015, 2018
  *
  */
 
+#include <getopt.h>
 #include <strings.h>
 #include <string.h>
 #include <stdlib.h>
 #include <stdio.h>
 
-struct facility_def {
+#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
+
+struct bitmap_def {
 	char *name;
 	int *bits;
 };
 
-static struct facility_def facility_defs[] = {
+static struct bitmap_def facility_defs[] = {
 	{
 		/*
 		 * FACILITIES_ALS contains the list of facilities that are
@@ -124,7 +125,7 @@ static struct facility_def facility_defs[] = {
 	},
 };
 
-static void print_facility_list(struct facility_def *def)
+static void print_bitmap(struct bitmap_def *def)
 {
 	unsigned int high, bit, dword, i;
 	unsigned long long *array;
@@ -151,25 +152,41 @@ static void print_facility_list(struct facility_def *def)
 	free(array);
 }
 
-static void print_facility_lists(void)
+static void print_header(const char *guard, struct bitmap_def *defs, size_t num)
 {
 	unsigned int i;
 
-	for (i = 0; i < sizeof(facility_defs) / sizeof(facility_defs[0]); i++)
-		print_facility_list(&facility_defs[i]);
-}
-
-int main(int argc, char **argv)
-{
-	printf("#ifndef __ASM_S390_FACILITY_DEFS__\n");
-	printf("#define __ASM_S390_FACILITY_DEFS__\n");
+	printf("#ifndef __ASM_S390_%s_DEFS__\n", guard);
+	printf("#define __ASM_S390_%s_DEFS__\n", guard);
 	printf("/*\n");
 	printf(" * DO NOT MODIFY.\n");
 	printf(" *\n");
 	printf(" * This file was generated by %s\n", __FILE__);
 	printf(" */\n\n");
 	printf("#include <linux/const.h>\n\n");
-	print_facility_lists();
+	for (i = 0; i < num; i++)
+		print_bitmap(&defs[i]);
 	printf("\n#endif\n");
+}
+
+static const struct option long_options[] = {
+	{ "facility", no_argument, NULL, 'f' },
+	{ NULL, 0, NULL, 0 }
+};
+
+int main(int argc, char **argv)
+{
+	int opt;
+
+	opt = getopt_long(argc, argv, "", long_options, NULL);
+	if (opt == -1)
+		return 1;
+	switch (opt) {
+	case 'f':
+		print_header("FACILITY", facility_defs, ARRAY_SIZE(facility_defs));
+		break;
+	default:
+		return 1;
+	}
 	return 0;
 }
-- 
2.53.0

[PATCH 06/32] s390/tools: Generate bitmap of nullified program checks

[Heiko Carstens]

Add PGM_CHECKS_NULLIFIED bitmap to gen_bitmaps and add a --pgmcheck
command line option to generate the pgmcheck-defs.h header file. The
bitmap specifies which program checks are nullified, and will be used
with future changes.

Change gen_bitmaps.c so it can create bitmaps with different bit
numbering: MSB0 bit numbering like before (most significant bit is
bit 0), and LSB0 bit numbering (least significant bit is bit 0),
which is used for the new bitmap.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/Kbuild  |  1 +
 arch/s390/tools/Makefile      |  8 +++++-
 arch/s390/tools/gen_bitmaps.c | 54 +++++++++++++++++++++++++++++++++--
 3 files changed, 59 insertions(+), 4 deletions(-)

diff --git a/arch/s390/include/asm/Kbuild b/arch/s390/include/asm/Kbuild
index 0c1fc47c3ba0..d13dd7e3dc01 100644
--- a/arch/s390/include/asm/Kbuild
+++ b/arch/s390/include/asm/Kbuild
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 generated-y += dis-defs.h
 generated-y += facility-defs.h
+generated-y += pgmcheck-defs.h
 generated-y += syscall_table.h
 generated-y += unistd_nr.h
 
diff --git a/arch/s390/tools/Makefile b/arch/s390/tools/Makefile
index ac267bb46550..ba1e42d005e7 100644
--- a/arch/s390/tools/Makefile
+++ b/arch/s390/tools/Makefile
@@ -4,7 +4,9 @@
 #
 
 kapi := arch/$(ARCH)/include/generated/asm
-kapi-hdrs-y := $(kapi)/facility-defs.h $(kapi)/dis-defs.h
+kapi-hdrs-y := $(kapi)/dis-defs.h
+kapi-hdrs-y += $(kapi)/facility-defs.h
+kapi-hdrs-y += $(kapi)/pgmcheck-defs.h
 
 PHONY += kapi
 
@@ -16,6 +18,7 @@ hostprogs		    += gen_opcode_table
 HOSTCFLAGS_gen_bitmaps.o += $(LINUXINCLUDE)
 
 filechk_facility-defs.h = $(obj)/gen_bitmaps --facility
+filechk_pgmcheck-defs.h = $(obj)/gen_bitmaps --pgmcheck
 
 filechk_dis-defs.h = \
 	$(obj)/gen_opcode_table < $(srctree)/arch/$(ARCH)/tools/opcodes.txt
@@ -23,6 +26,9 @@ filechk_dis-defs.h = \
 $(kapi)/facility-defs.h: $(obj)/gen_bitmaps FORCE
 	$(call filechk,facility-defs.h)
 
+$(kapi)/pgmcheck-defs.h: $(obj)/gen_bitmaps FORCE
+	$(call filechk,pgmcheck-defs.h)
+
 $(kapi)/dis-defs.h: $(obj)/gen_opcode_table FORCE
 	$(call filechk,dis-defs.h)
 
diff --git a/arch/s390/tools/gen_bitmaps.c b/arch/s390/tools/gen_bitmaps.c
index 4bc5f19dd405..2e5dc491b19e 100644
--- a/arch/s390/tools/gen_bitmaps.c
+++ b/arch/s390/tools/gen_bitmaps.c
@@ -17,6 +17,8 @@
 struct bitmap_def {
 	char *name;
 	int *bits;
+	int inverted;
+	int maxbit;
 };
 
 static struct bitmap_def facility_defs[] = {
@@ -27,6 +29,7 @@ static struct bitmap_def facility_defs[] = {
 		 * -march=<machine>.
 		 */
 		.name = "FACILITIES_ALS",
+		.inverted = 1,
 		.bits = (int[]){
 			0,  /* N3 instructions */
 			1,  /* z/Arch mode installed */
@@ -71,6 +74,7 @@ static struct bitmap_def facility_defs[] = {
 		 * guest facilities supported by KVM.
 		 */
 		.name = "FACILITIES_KVM",
+		.inverted = 1,
 		.bits = (int[]){
 			0,  /* N3 instructions */
 			1,  /* z/Arch mode installed */
@@ -116,6 +120,7 @@ static struct bitmap_def facility_defs[] = {
 		 */
 
 		.name = "FACILITIES_KVM_CPUMODEL",
+		.inverted = 1,
 		.bits = (int[]){
 			12, /* AP Query Configuration Information */
 			15, /* AP Facilities Test */
@@ -125,17 +130,56 @@ static struct bitmap_def facility_defs[] = {
 	},
 };
 
+static struct bitmap_def pgm_check_defs[] = {
+	{
+		.name = "PGM_CHECKS_NULLIFIED",
+		.inverted = 0,
+		.maxbit = 127,
+		.bits = (int[]){
+			0x10, /* Segment Translation */
+			0x11, /* Page Translation */
+			0x16, /* Trace Table */
+			0x20, /* AFX Translation */
+			0x21, /* ASX Translation */
+			0x22, /* LX Translation */
+			0x23, /* EX Translation */
+			0x24, /* Primary Authority */
+			0x25, /* Secondary Authority */
+			0x26, /* LFX Translation */
+			0x27, /* LSX Translation */
+			0x29, /* ALEN Translation */
+			0x2a, /* ALE Sequence */
+			0x2b, /* ASTE Validity */
+			0x2c, /* ASTE Sequence */
+			0x2d, /* Extended Authority */
+			0x2e, /* LSTE Sequence */
+			0x2f, /* ASTE Instance */
+			0x30, /* Stack Full */
+			0x31, /* Stack Empty */
+			0x32, /* Stack Specification */
+			0x33, /* Stack Type */
+			0x34, /* Stack Operation */
+			0x38, /* ASCE Type */
+			0x39, /* Region First Translation */
+			0x3a, /* Region Second Translation */
+			0x3b, /* Region Third Translation */
+			0x3d, /* Secure Storage Access */
+			-1  /* END */
+		}
+	},
+};
+
 static void print_bitmap(struct bitmap_def *def)
 {
 	unsigned int high, bit, dword, i;
 	unsigned long long *array;
 
-	array = calloc(1, 8);
+	high = def->maxbit ? def->maxbit / 64 : 0;
+	array = calloc(high + 1, 8);
 	if (!array)
 		exit(EXIT_FAILURE);
-	high = 0;
 	for (i = 0; def->bits[i] != -1; i++) {
-		bit = 63 - (def->bits[i] & 63);
+		bit = def->inverted ? 63 - (def->bits[i] & 63) : (def->bits[i] & 63);
 		dword = def->bits[i] / 64;
 		if (dword > high) {
 			array = realloc(array, (dword + 1) * 8);
@@ -171,6 +215,7 @@ static void print_header(const char *guard, struct bitmap_def *defs, size_t num)
 
 static const struct option long_options[] = {
 	{ "facility", no_argument, NULL, 'f' },
+	{ "pgmcheck", no_argument, NULL, 'p' },
 	{ NULL, 0, NULL, 0 }
 };
 
@@ -185,6 +230,9 @@ int main(int argc, char **argv)
 	case 'f':
 		print_header("FACILITY", facility_defs, ARRAY_SIZE(facility_defs));
 		break;
+	case 'p':
+		print_header("PGMCHECK", pgm_check_defs, ARRAY_SIZE(pgm_check_defs));
+		break;
 	default:
 		return 1;
 	}
-- 
2.53.0

[PATCH 07/32] s390/traps: Handle nullifying program checks in get_trap_ip()

[Heiko Carstens]

get_trap_ip() always rewinds the PSW address by the instruction length
from the interruption code. For nullifying program checks the PSW
already points at the faulting instruction, so rewinding is wrong.

Add a nullified[] bitmap and return the unchanged PSW address for
nullifying cases, or if PIF_PSW_ADDR_ADJUSTED is set.

No functional change for existing callers, which only deal with
suppressing, terminating, or completing program checks.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/traps.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/arch/s390/kernel/traps.c b/arch/s390/kernel/traps.c
index bfcab4592299..5e63032a56db 100644
--- a/arch/s390/kernel/traps.c
+++ b/arch/s390/kernel/traps.c
@@ -26,6 +26,7 @@
 #include <linux/entry-common.h>
 #include <linux/kmsan.h>
 #include <linux/bug.h>
+#include <asm/pgmcheck-defs.h>
 #include <asm/entry-percpu.h>
 #include <asm/asm-extable.h>
 #include <asm/irqflags.h>
@@ -41,15 +42,19 @@ struct pgm_stat {
 
 static DEFINE_PER_CPU_SHARED_ALIGNED(struct pgm_stat, pgm_stat);
 
-static inline void __user *get_trap_ip(struct pt_regs *regs)
+static inline unsigned long get_trap_ip(struct pt_regs *regs)
 {
-	unsigned long address;
+	static const unsigned long nullified[2] = { PGM_CHECKS_NULLIFIED };
+	psw_t psw = regs->psw;
 
+	if (test_pt_regs_flag(regs, PIF_PSW_ADDR_ADJUSTED))
+		return psw.addr;
 	if (regs->int_code & 0x200)
-		address = current->thread.trap_tdb.data[3];
-	else
-		address = regs->psw.addr;
-	return (void __user *)(address - (regs->int_code >> 16));
+		psw.addr = current->thread.trap_tdb.data[3];
+	if (test_bit(regs->int_code & PGM_INT_CODE_MASK, nullified))
+		return psw.addr;
+	psw.addr = __rewind_psw(psw, regs->int_code >> 16);
+	return psw.addr;
 }
 
 #ifdef CONFIG_GENERIC_BUG
@@ -62,7 +67,7 @@ int is_valid_bugaddr(unsigned long addr)
 void do_report_trap(struct pt_regs *regs, int si_signo, int si_code, char *str)
 {
 	if (user_mode(regs)) {
-		force_sig_fault(si_signo, si_code, get_trap_ip(regs));
+		force_sig_fault(si_signo, si_code, (void __user *)get_trap_ip(regs));
 		report_user_fault(regs, si_signo, 0);
 	} else {
 		if (!fixup_exception(regs))
@@ -152,7 +157,7 @@ static void illegal_op(struct pt_regs *regs)
 	int signal = 0;
 	u16 opcode;
 
-	location = get_trap_ip(regs);
+	location = (void __user *)get_trap_ip(regs);
 	if (user_mode(regs)) {
 		if (get_user(opcode, location))
 			return;
-- 
2.53.0

[PATCH 08/32] s390/dis: Improve show_code() output

[Heiko Carstens]

Replace the '>' and '*' instruction markers with explicit 'PSW >' and
'PGM >' markers which identify both the PSW address and the faulting
instruction.

Previously, '>' marked the PSW address, while '*' marked the preceding
instruction as a guess for the faulting instruction for suppressing
exceptions:

 Krnl Code:
            000003ff8dc7bff8: e310f0080004 lg      %r1,8(%r15)
           *000003ff8dc7bffe: e3a0e0000004 lg      %r10,0(%r14)
           >000003ff8dc7c004: a7280000     lhi     %r2,0
            000003ff8dc7c008: eb9ff0900024 stmg    %r9,%r15,144(%r15)

Now 'PGM >' explicitly marks the faulting instruction without guessing
and 'PSW >' marks the PSW address:

 Krnl Code:
            000003ff8dc7bff8: e310f0080004 lg      %r1,8(%r15)
       PGM >000003ff8dc7bffe: e3a0e0000004 lg      %r10,0(%r14)
       PSW >000003ff8dc7c004: a7280000     lhi     %r2,0
            000003ff8dc7c008: eb9ff0900024 stmg    %r9,%r15,144(%r15)

For nullifying exceptions where both addresses are the same,
'PGM+PSW >' is printed instead:

 Krnl Code:
            000003ff8dc7bff8: e310f0080004 lg      %r1,8(%r15)
   PGM+PSW >000003ff8dc7bffe: e3a0e0000004 lg      %r10,0(%r14)
            000003ff8dc7c004: a7280000     lhi     %r2,0
            000003ff8dc7c008: eb9ff0900024 stmg    %r9,%r15,144(%r15)

show_code() may also be called in other contexts where pt_regs does not
originate from a program check (e.g. system calls). Introduce and set
PIF_PGM_CHECK to indicate when in program check context, and print the
PGM marker only in such a context.

Note that corner cases are not taken into account. Like before kprobes is
not considered, which means that the disassembly may contain unexpected
instructions, or that disassembly and/or instruction length count as well
as interruption do not match.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/ptrace.h |  4 ++++
 arch/s390/kernel/dis.c         | 16 +++++++++++-----
 arch/s390/kernel/early.c       |  1 +
 arch/s390/kernel/traps.c       |  4 +++-
 4 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/arch/s390/include/asm/ptrace.h b/arch/s390/include/asm/ptrace.h
index c844b8aa0fd9..3e56ef8c268b 100644
--- a/arch/s390/include/asm/ptrace.h
+++ b/arch/s390/include/asm/ptrace.h
@@ -18,12 +18,14 @@
 #define PIF_SYSCALL_RET_SET		2	/* return value was set via ptrace */
 #define PIF_FTRACE_FULL_REGS		3	/* all register contents valid (ftrace) */
 #define PIF_WARN_TRAP_SKIPPED		4	/* __WARN_trap() was skipped */
+#define PIF_PGM_CHECK			5	/* inside a program check */
 
 #define _PIF_SYSCALL			BIT(PIF_SYSCALL)
 #define _PIF_ADDR_PSW_ADJUSTED		BIT(PIF_PSW_ADDR_ADJUSTED)
 #define _PIF_SYSCALL_RET_SET		BIT(PIF_SYSCALL_RET_SET)
 #define _PIF_FTRACE_FULL_REGS		BIT(PIF_FTRACE_FULL_REGS)
 #define _PIF_WARN_TRAP_SKIPPED		BIT(PIF_WARN_TRAP_SKIPPED)
+#define _PIF_PGM_CHECK			BIT(PIF_PGM_CHECK)
 
 #define PSW32_MASK_PER		_AC(0x40000000, UL)
 #define PSW32_MASK_DAT		_AC(0x04000000, UL)
@@ -242,6 +244,8 @@ static inline void instruction_pointer_set(struct pt_regs *regs,
 	regs->psw.addr = val;
 }
 
+unsigned long get_trap_ip(struct pt_regs *regs);
+
 int regs_query_register_offset(const char *name);
 const char *regs_query_register_name(unsigned int offset);
 
diff --git a/arch/s390/kernel/dis.c b/arch/s390/kernel/dis.c
index 5a1a24d211c3..ece0b1e46ad4 100644
--- a/arch/s390/kernel/dis.c
+++ b/arch/s390/kernel/dis.c
@@ -503,7 +503,8 @@ static int copy_from_regs(struct pt_regs *regs, void *dst, void *src, int len)
 void show_code(struct pt_regs *regs)
 {
 	char *mode = user_mode(regs) ? "User" : "Krnl";
-	unsigned long addr, pswaddr;
+	unsigned long addr, pswaddr, trapaddr;
+	bool pgm_check;
 	unsigned char code[64];
 	char buffer[128], *ptr;
 	int start, end, opsize, hops, i;
@@ -511,6 +512,9 @@ void show_code(struct pt_regs *regs)
 	pswaddr = regs->psw.addr;
 	if (test_pt_regs_flag(regs, PIF_PSW_ADDR_ADJUSTED))
 		pswaddr = __forward_psw(regs->psw, regs->int_code >> 16);
+	pgm_check = test_pt_regs_flag(regs, PIF_PGM_CHECK);
+	if (pgm_check)
+		trapaddr = get_trap_ip(regs);
 	/* Get a snapshot of the 64 bytes surrounding the fault address. */
 	for (start = 32; start && pswaddr >= 34 - start; start -= 2) {
 		addr = pswaddr - 34 + start;
@@ -545,15 +549,17 @@ void show_code(struct pt_regs *regs)
 	while (start < end && hops < 8) {
 		ptr = buffer;
 		opsize = insn_length(code[start]);
+		addr = pswaddr + start - 32;
 		if (start == 32 && test_pt_regs_flag(regs, PIF_WARN_TRAP_SKIPPED))
 			ptr += sprintf(ptr, " WARN RET >");
-		else if (start + opsize == 32 && !test_pt_regs_flag(regs, PIF_WARN_TRAP_SKIPPED))
-			ptr += sprintf(ptr, "          *");
+		else if (pgm_check && addr == trapaddr && start == 32)
+			ptr += sprintf(ptr, "  PGM+PSW >");
+		else if (pgm_check && addr == trapaddr)
+			ptr += sprintf(ptr, "      PGM >");
 		else if (start == 32)
-			ptr += sprintf(ptr, "          >");
+			ptr += sprintf(ptr, "      PSW >");
 		else
 			ptr += sprintf(ptr, "           ");
-		addr = pswaddr + start - 32;
 		ptr += sprintf(ptr, "%px: ", (void *)addr);
 		if (start + opsize >= end)
 			break;
diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c
index d875b3b304be..a603e034fa82 100644
--- a/arch/s390/kernel/early.c
+++ b/arch/s390/kernel/early.c
@@ -159,6 +159,7 @@ void __init __do_early_pgm_check(struct pt_regs *regs)
 	regs->int_parm_long = lc->trans_exc_code;
 	regs->last_break = lc->pgm_last_break;
 	regs->monitor_code = lc->monitor_code;
+	set_pt_regs_flag(regs, PIF_PGM_CHECK);
 
 	/* Monitor Event? Might be a warning */
 	if ((regs->int_code & PGM_INT_CODE_MASK) == 0x40) {
diff --git a/arch/s390/kernel/traps.c b/arch/s390/kernel/traps.c
index 5e63032a56db..a0937b4a6cf3 100644
--- a/arch/s390/kernel/traps.c
+++ b/arch/s390/kernel/traps.c
@@ -42,7 +42,7 @@ struct pgm_stat {
 
 static DEFINE_PER_CPU_SHARED_ALIGNED(struct pgm_stat, pgm_stat);
 
-static inline unsigned long get_trap_ip(struct pt_regs *regs)
+unsigned long get_trap_ip(struct pt_regs *regs)
 {
 	static const unsigned long nullified[2] = { PGM_CHECKS_NULLIFIED };
 	psw_t psw = regs->psw;
@@ -56,6 +56,7 @@ static inline unsigned long get_trap_ip(struct pt_regs *regs)
 	psw.addr = __rewind_psw(psw, regs->int_code >> 16);
 	return psw.addr;
 }
+NOKPROBE_SYMBOL(get_trap_ip);
 
 #ifdef CONFIG_GENERIC_BUG
 int is_valid_bugaddr(unsigned long addr)
@@ -354,6 +355,7 @@ void noinstr __do_pgm_check(struct pt_regs *regs, unsigned long flags)
 	regs->int_code = lc->pgm_int_code;
 	regs->int_parm_long = teid.val;
 	regs->monitor_code = lc->monitor_code;
+	set_pt_regs_flag(regs, PIF_PGM_CHECK);
 
 	trapnr = regs->int_code & PGM_INT_CODE_MASK;
 	stat = this_cpu_ptr(&pgm_stat);
-- 
2.53.0

[PATCH 09/32] s390/extable: Rework exception handling logic

[Heiko Carstens]

Add EX_TABLE_INSN macros which will replace the existing EX_TABLE
macros. The macros come with new semantics for the whole exception
handling: with the new exception handling the exception table entries
always point to the instruction which may cause an exception. This is
regardless of exception type, and whether an exception is nullifying,
suppressing, completing, or terminating.

This improves the current situation where exception table entries are
required which point to the instruction (nullifying) and/or behind the
instruction (suppressing, completing, or terminating), since depending
on usage an instruction can generate different exceptions.

The new EX_TABLE_INSN macros are aliases to the existing EX_TABLE
macros. The exception handling logic is changed to first perform a lookup
in the exception tables with get_trap_ip() / boot_get_trap_ip(). If that
finds nothing, it falls back to looking up with instruction_pointer().

This is a temporary solution until all EX_TABLE usages have been
converted to EX_TABLE_INSN. When that work is complete EX_TABLE will
be removed, exception tables will only be searched by using the return
value of get_trap_ip() / boot_get_trap_ip().

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/boot/pgm_check.c          | 35 ++++++++++++++++++++++++-----
 arch/s390/include/asm/asm-extable.h | 10 +++++++++
 arch/s390/mm/extable.c              |  4 +++-
 3 files changed, 42 insertions(+), 7 deletions(-)

diff --git a/arch/s390/boot/pgm_check.c b/arch/s390/boot/pgm_check.c
index fa621fa5bc02..56d0be7e2823 100644
--- a/arch/s390/boot/pgm_check.c
+++ b/arch/s390/boot/pgm_check.c
@@ -3,9 +3,11 @@
 #include <linux/stdarg.h>
 #include <linux/string.h>
 #include <linux/ctype.h>
+#include <asm/pgmcheck-defs.h>
 #include <asm/stacktrace.h>
 #include <asm/boot_data.h>
 #include <asm/lowcore.h>
+#include <asm/processor.h>
 #include <asm/setup.h>
 #include <asm/sclp.h>
 #include <asm/uv.h>
@@ -40,19 +42,40 @@ static inline unsigned long extable_insn(const struct exception_table_entry *x)
 	return (unsigned long)&x->insn + x->insn;
 }
 
-static bool ex_handler(struct pt_regs *regs)
+static unsigned long boot_get_trap_ip(struct pt_regs *regs)
+{
+	static const unsigned long nullified[2] = { PGM_CHECKS_NULLIFIED };
+
+	if (test_bit(regs->int_code & PGM_INT_CODE_MASK, nullified))
+		return regs->psw.addr;
+	return __rewind_psw(regs->psw, regs->int_code >> 16);
+}
+
+static const struct exception_table_entry *search_boot_extable(unsigned long ip)
 {
 	const struct exception_table_entry *ex;
 
 	for (ex = __start___ex_table; ex < __stop___ex_table; ex++) {
-		if (extable_insn(ex) != regs->psw.addr)
+		if (extable_insn(ex) != ip)
 			continue;
 		if (ex->type != EX_TYPE_FIXUP)
-			return false;
-		regs->psw.addr = extable_fixup(ex);
-		return true;
+			return NULL;
+		return ex;
 	}
-	return false;
+	return NULL;
+}
+
+static bool ex_handler(struct pt_regs *regs)
+{
+	const struct exception_table_entry *ex;
+
+	ex = search_boot_extable(boot_get_trap_ip(regs));
+	if (!ex)
+		ex = search_boot_extable(instruction_pointer(regs));
+	if (!ex)
+		return false;
+	regs->psw.addr = extable_fixup(ex);
+	return true;
 }
 
 void do_pgm_check(struct pt_regs *regs)
diff --git a/arch/s390/include/asm/asm-extable.h b/arch/s390/include/asm/asm-extable.h
index d23ea0c94e4e..488bd10452b3 100644
--- a/arch/s390/include/asm/asm-extable.h
+++ b/arch/s390/include/asm/asm-extable.h
@@ -92,4 +92,14 @@
 #define EX_TABLE_UA_MVCOS_FROM(_fault, _target)				\
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_UA_MVCOS_FROM, __stringify(%%r0), __stringify(%%r0), 0)
 
+#define EX_TABLE_INSN				EX_TABLE
+#define EX_TABLE_INSN_AMODE31			EX_TABLE_AMODE31
+#define EX_TABLE_INSN_UA_FAULT			EX_TABLE_UA_FAULT
+#define EX_TABLE_INSN_UA_LOAD_REG		EX_TABLE_UA_LOAD_REG
+#define EX_TABLE_INSN_UA_LOAD_REGPAIR		EX_TABLE_UA_LOAD_REGPAIR
+#define EX_TABLE_INSN_ZEROPAD			EX_TABLE_ZEROPAD
+#define EX_TABLE_INSN_FPC			EX_TABLE_FPC
+#define EX_TABLE_INSN_UA_MVCOS_TO		EX_TABLE_UA_MVCOS_TO
+#define EX_TABLE_INSN_UA_MVCOS_FROM		EX_TABLE_UA_MVCOS_FROM
+
 #endif /* __ASM_EXTABLE_H */
diff --git a/arch/s390/mm/extable.c b/arch/s390/mm/extable.c
index 7498e858c401..ca6a1efa49ee 100644
--- a/arch/s390/mm/extable.c
+++ b/arch/s390/mm/extable.c
@@ -120,7 +120,9 @@ bool fixup_exception(struct pt_regs *regs)
 {
 	const struct exception_table_entry *ex;
 
-	ex = s390_search_extables(instruction_pointer(regs));
+	ex = s390_search_extables(get_trap_ip(regs));
+	if (!ex)
+		ex = s390_search_extables(instruction_pointer(regs));
 	if (!ex)
 		return false;
 	switch (ex->type) {
-- 
2.53.0

[PATCH 10/32] s390/diag: Convert amode31 exception tables to EX_TABLE_INSN

[Heiko Carstens]

Convert all amode31 exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so named labels are replaced with numeric labels.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/text_amode31.S | 34 ++++++++++++++-------------------
 1 file changed, 14 insertions(+), 20 deletions(-)

diff --git a/arch/s390/kernel/text_amode31.S b/arch/s390/kernel/text_amode31.S
index f007d892d0c5..7761de491443 100644
--- a/arch/s390/kernel/text_amode31.S
+++ b/arch/s390/kernel/text_amode31.S
@@ -19,9 +19,9 @@
  * affects a few functions that are not performance-relevant.
  */
 	.macro BR_EX_AMODE31_r14
-	exrl	0,0f
+	exrl	0,555f
 	j	.
-0:	br	%r14
+555:	br	%r14
 	.endm
 
 /*
@@ -33,15 +33,13 @@ SYM_TYPED_FUNC_START(_diag14_amode31)
 	lgr	%r3,%r4
 	lhi	%r5,-EIO
 	sam31
-	diag	%r1,%r2,0x14
-.Ldiag14_ex:
+0:	diag	%r1,%r2,0x14
 	ipm	%r5
 	srl	%r5,28
-.Ldiag14_fault:
-	sam64
+1:	sam64
 	lgfr	%r2,%r5
 	BR_EX_AMODE31_r14
-	EX_TABLE_AMODE31(.Ldiag14_ex, .Ldiag14_fault)
+	EX_TABLE_INSN_AMODE31(0b, 1b)
 SYM_FUNC_END(_diag14_amode31)
 
 /*
@@ -51,15 +49,13 @@ SYM_TYPED_FUNC_START(_diag210_amode31)
 	lgr	%r1,%r2
 	lhi	%r2,-1
 	sam31
-	diag	%r1,%r0,0x210
-.Ldiag210_ex:
+0:	diag	%r1,%r0,0x210
 	ipm	%r2
 	srl	%r2,28
-.Ldiag210_fault:
-	sam64
+1:	sam64
 	lgfr	%r2,%r2
 	BR_EX_AMODE31_r14
-	EX_TABLE_AMODE31(.Ldiag210_ex, .Ldiag210_fault)
+	EX_TABLE_INSN_AMODE31(0b, 1b)
 SYM_FUNC_END(_diag210_amode31)
 
 /*
@@ -68,12 +64,11 @@ SYM_FUNC_END(_diag210_amode31)
 SYM_TYPED_FUNC_START(_diag8c_amode31)
 	llgf	%r3,0(%r3)
 	sam31
-	diag	%r2,%r4,0x8c
-.Ldiag8c_ex:
-	sam64
+0:	diag	%r2,%r4,0x8c
+1:	sam64
 	lgfr	%r2,%r3
 	BR_EX_AMODE31_r14
-	EX_TABLE_AMODE31(.Ldiag8c_ex, .Ldiag8c_ex)
+	EX_TABLE_INSN_AMODE31(0b, 1b)
 SYM_FUNC_END(_diag8c_amode31)
 /*
  * int _diag26c_amode31(void *req, void *resp, enum diag26c_sc subcode)
@@ -81,12 +76,11 @@ SYM_FUNC_END(_diag8c_amode31)
 SYM_TYPED_FUNC_START(_diag26c_amode31)
 	lghi	%r5,-EOPNOTSUPP
 	sam31
-	diag	%r2,%r4,0x26c
-.Ldiag26c_ex:
-	sam64
+0:	diag	%r2,%r4,0x26c
+1:	sam64
 	lgfr	%r2,%r5
 	BR_EX_AMODE31_r14
-	EX_TABLE_AMODE31(.Ldiag26c_ex, .Ldiag26c_ex)
+	EX_TABLE_INSN_AMODE31(0b, 1b)
 SYM_FUNC_END(_diag26c_amode31)
 
 /*
-- 
2.53.0

[PATCH 11/32] s390/uaccess: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all uaccess exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so duplicate entries and nopr padding are no longer needed and removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/uaccess.h | 45 +++++++++++----------------------
 1 file changed, 15 insertions(+), 30 deletions(-)

diff --git a/arch/s390/include/asm/uaccess.h b/arch/s390/include/asm/uaccess.h
index a9f32c53f699..ac99b32c6ceb 100644
--- a/arch/s390/include/asm/uaccess.h
+++ b/arch/s390/include/asm/uaccess.h
@@ -43,10 +43,8 @@ raw_copy_from_user(void *to, const void __user *from, unsigned long size)
 		asm_inline volatile(
 			"	lhi	%%r0,%[spec]\n"
 			"0:	mvcos	%[to],%[from],%[size]\n"
-			"1:	nopr	%%r7\n"
 			CC_IPM(cc)
-			EX_TABLE_UA_MVCOS_FROM(0b, 0b)
-			EX_TABLE_UA_MVCOS_FROM(1b, 0b)
+			EX_TABLE_INSN_UA_MVCOS_FROM(0b, 0b)
 			: CC_OUT(cc, cc), [size] "+d" (size), [to] "=Q" (*(char *)to)
 			: [spec] "I" (0x81), [from] "Q" (*(const char __user *)from)
 			: CC_CLOBBER_LIST("memory", "0"));
@@ -71,10 +69,8 @@ raw_copy_to_user(void __user *to, const void *from, unsigned long size)
 		asm_inline volatile(
 			"	llilh	%%r0,%[spec]\n"
 			"0:	mvcos	%[to],%[from],%[size]\n"
-			"1:	nopr	%%r7\n"
 			CC_IPM(cc)
-			EX_TABLE_UA_MVCOS_TO(0b, 0b)
-			EX_TABLE_UA_MVCOS_TO(1b, 0b)
+			EX_TABLE_INSN_UA_MVCOS_TO(0b, 0b)
 			: CC_OUT(cc, cc), [size] "+d" (size), [to] "=Q" (*(char __user *)to)
 			: [spec] "I" (0x81), [from] "Q" (*(const char *)from)
 			: CC_CLOBBER_LIST("memory", "0"));
@@ -123,9 +119,7 @@ __put_user_##type##_noinstr(unsigned type __user *to,			\
 	asm goto(							\
 		"	llilh	%%r0,%[spec]\n"				\
 		"0:	mvcos	%[to],%[from],%[size]\n"		\
-		"1:	nopr	%%r7\n"					\
-		EX_TABLE(0b, %l[Efault])				\
-		EX_TABLE(1b, %l[Efault])				\
+		EX_TABLE_INSN(0b, %l[Efault])				\
 		: [to] "+Q" (*to)					\
 		: [size] "d" (size), [from] "Q" (*from),		\
 		  [spec] "I" (0x81)					\
@@ -150,10 +144,9 @@ __put_user_##type##_noinstr(unsigned type __user *to,			\
 	asm_inline volatile(						\
 		"	llilh	%%r0,%[spec]\n"				\
 		"0:	mvcos	%[to],%[from],%[size]\n"		\
-		"1:	lhi	%[rc],0\n"				\
-		"2:\n"							\
-		EX_TABLE_UA_FAULT(0b, 2b, %[rc])			\
-		EX_TABLE_UA_FAULT(1b, 2b, %[rc])			\
+		"	lhi	%[rc],0\n"				\
+		"1:\n"							\
+		EX_TABLE_INSN_UA_FAULT(0b, 1b, %[rc])			\
 		: [rc] "=d" (rc), [to] "+Q" (*to)			\
 		: [size] "d" (size), [from] "Q" (*from),		\
 		  [spec] "I" (0x81)					\
@@ -238,9 +231,7 @@ __get_user_##type##_noinstr(unsigned type *to,				\
 	asm goto(							\
 		"	lhi	%%r0,%[spec]\n"				\
 		"0:	mvcos	%[to],%[from],%[size]\n"		\
-		"1:	nopr	%%r7\n"					\
-		EX_TABLE(0b, %l[Efault])				\
-		EX_TABLE(1b, %l[Efault])				\
+		EX_TABLE_INSN(0b, %l[Efault])				\
 		: [to] "=Q" (*to)					\
 		: [size] "d" (size), [from] "Q" (*from),		\
 		  [spec] "I" (0x81)					\
@@ -266,10 +257,9 @@ __get_user_##type##_noinstr(unsigned type *to,				\
 	asm_inline volatile(						\
 		"	lhi	%%r0,%[spec]\n"				\
 		"0:	mvcos	%[to],%[from],%[size]\n"		\
-		"1:	lhi	%[rc],0\n"				\
-		"2:\n"							\
-		EX_TABLE_UA_FAULT(0b, 2b, %[rc])			\
-		EX_TABLE_UA_FAULT(1b, 2b, %[rc])			\
+		"	lhi	%[rc],0\n"				\
+		"1:\n"							\
+		EX_TABLE_INSN_UA_FAULT(0b, 1b, %[rc])			\
 		: [rc] "=d" (rc), [to] "=Q" (*to)			\
 		: [size] "d" (size), [from] "Q" (*from),		\
 		  [spec] "I" (0x81)					\
@@ -374,10 +364,8 @@ __clear_user(void __user *to, unsigned long size)
 		asm_inline volatile(
 			"	llilh	%%r0,%[spec]\n"
 			"0:	mvcos	%[to],%[from],%[size]\n"
-			"1:	nopr	%%r7\n"
 			CC_IPM(cc)
-			EX_TABLE_UA_MVCOS_TO(0b, 0b)
-			EX_TABLE_UA_MVCOS_TO(1b, 0b)
+			EX_TABLE_INSN_UA_MVCOS_TO(0b, 0b)
 			: CC_OUT(cc, cc), [size] "+d" (size), [to] "=Q" (*(char __user *)to)
 			: [spec] "I" (0x81), [from] "Q" (*(const char *)empty_zero_page)
 			: CC_CLOBBER_LIST("memory", "0"));
@@ -418,9 +406,7 @@ do {									\
 	case 8:								\
 		asm goto(						\
 			"0:	mvc	%O[_dst](%[_len],%R[_dst]),%[_src]\n" \
-			"1:	nopr	%%r7\n"				\
-			EX_TABLE(0b, %l[err_label])			\
-			EX_TABLE(1b, %l[err_label])			\
+			EX_TABLE_INSN(0b, %l[err_label])		\
 			: [_dst] "=Q" (*(type *)dst)			\
 			: [_src] "Q" (*(type *)(src)),			\
 			  [_len] "I" (sizeof(type))			\
@@ -447,10 +433,9 @@ do {									\
 	case 8:								\
 		asm_inline volatile(					\
 			"0:	mvc	0(%[_len],%[_dst]),%[_src]\n"	\
-			"1:	lhi	%[_rc],0\n"			\
-			"2:\n"						\
-			EX_TABLE_UA_FAULT(0b, 2b, %[_rc])		\
-			EX_TABLE_UA_FAULT(1b, 2b, %[_rc])		\
+			"	lhi	%[_rc],0\n"			\
+			"1:\n"						\
+			EX_TABLE_INSN_UA_FAULT(0b, 1b, %[_rc])		\
 			: [_rc] "=d" (__rc),				\
 			  "=m" (*__dst)					\
 			: [_src] "Q" (*(type *)(src)),			\
-- 
2.53.0

[PATCH 12/32] s390/lib/uaccess: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all uaccess exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so duplicate entries are removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/lib/uaccess.c | 29 ++++++++++++-----------------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/arch/s390/lib/uaccess.c b/arch/s390/lib/uaccess.c
index 0ac2f3998b14..d3f462c12bba 100644
--- a/arch/s390/lib/uaccess.c
+++ b/arch/s390/lib/uaccess.c
@@ -49,24 +49,22 @@ static nokprobe_inline int __cmpxchg_key_small(void *address, unsigned int *uval
 		"20:	spka	0(%[key])\n"
 		"	llill	%[count],%[max_loops]\n"
 		"0:	l	%[prev],%[address]\n"
-		"1:	nr	%[prev],%[mask]\n"
+		"	nr	%[prev],%[mask]\n"
 		"	xilf	%[mask],0xffffffff\n"
 		"	or	%[new],%[prev]\n"
 		"	or	%[prev],%[tmp]\n"
-		"2:	lr	%[tmp],%[prev]\n"
-		"3:	cs	%[prev],%[new],%[address]\n"
-		"4:	jnl	5f\n"
+		"1:	lr	%[tmp],%[prev]\n"
+		"2:	cs	%[prev],%[new],%[address]\n"
+		"	jnl	3f\n"
 		"	xr	%[tmp],%[prev]\n"
 		"	xr	%[new],%[tmp]\n"
 		"	nr	%[tmp],%[mask]\n"
-		"	jnz	5f\n"
-		"	brct	%[count],2b\n"
-		"5:	spka	%[default_key]\n"
+		"	jnz	3f\n"
+		"	brct	%[count],1b\n"
+		"3:	spka	%[default_key]\n"
 		"21:\n"
-		EX_TABLE_UA_LOAD_REG(0b, 5b, %[rc], %[prev])
-		EX_TABLE_UA_LOAD_REG(1b, 5b, %[rc], %[prev])
-		EX_TABLE_UA_LOAD_REG(3b, 5b, %[rc], %[prev])
-		EX_TABLE_UA_LOAD_REG(4b, 5b, %[rc], %[prev])
+		EX_TABLE_INSN_UA_LOAD_REG(0b, 3b, %[rc], %[prev])
+		EX_TABLE_INSN_UA_LOAD_REG(2b, 3b, %[rc], %[prev])
 		SKEY_REGION(20b, 21b)
 		: [rc] "+&d" (rc),
 		[prev] "=&d" (prev),
@@ -133,8 +131,7 @@ int __kprobes __cmpxchg_key4(void *address, unsigned int *uval, unsigned int old
 		"0:	cs	%[prev],%[new],%[address]\n"
 		"1:	spka	%[default_key]\n"
 		"21:\n"
-		EX_TABLE_UA_LOAD_REG(0b, 1b, %[rc], %[prev])
-		EX_TABLE_UA_LOAD_REG(1b, 1b, %[rc], %[prev])
+		EX_TABLE_INSN_UA_LOAD_REG(0b, 1b, %[rc], %[prev])
 		SKEY_REGION(20b, 21b)
 		: [rc] "+&d" (rc),
 		[prev] "+&d" (prev),
@@ -160,8 +157,7 @@ int __kprobes __cmpxchg_key8(void *address, unsigned long *uval, unsigned long o
 		"0:	csg	%[prev],%[new],%[address]\n"
 		"1:	spka	%[default_key]\n"
 		"21:\n"
-		EX_TABLE_UA_LOAD_REG(0b, 1b, %[rc], %[prev])
-		EX_TABLE_UA_LOAD_REG(1b, 1b, %[rc], %[prev])
+		EX_TABLE_INSN_UA_LOAD_REG(0b, 1b, %[rc], %[prev])
 		SKEY_REGION(20b, 21b)
 		: [rc] "+&d" (rc),
 		[prev] "+&d" (prev),
@@ -187,8 +183,7 @@ int __kprobes __cmpxchg_key16(void *address, __uint128_t *uval, __uint128_t old,
 		"0:	cdsg	%[prev],%[new],%[address]\n"
 		"1:	spka	%[default_key]\n"
 		"21:\n"
-		EX_TABLE_UA_LOAD_REGPAIR(0b, 1b, %[rc], %[prev])
-		EX_TABLE_UA_LOAD_REGPAIR(1b, 1b, %[rc], %[prev])
+		EX_TABLE_INSN_UA_LOAD_REGPAIR(0b, 1b, %[rc], %[prev])
 		SKEY_REGION(20b, 21b)
 		: [rc] "+&d" (rc),
 		[prev] "+&d" (prev),
-- 
2.53.0

[PATCH 13/32] s390/futex: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all futex exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so duplicate entries are removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/futex.h | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/arch/s390/include/asm/futex.h b/arch/s390/include/asm/futex.h
index 942f21c39697..71e8060637be 100644
--- a/arch/s390/include/asm/futex.h
+++ b/arch/s390/include/asm/futex.h
@@ -23,13 +23,11 @@ __futex_atomic_##name(int oparg, int *old, u32 __user *uaddr)			\
 		"0:	l	%[old],%[uaddr]\n"				\
 		"1:"insn							\
 		"2:	cs	%[old],%[new],%[uaddr]\n"			\
-		"3:	jl	1b\n"						\
+		"	jl	1b\n"						\
 		"	lhi	%[rc],0\n"					\
-		"4:	sacf	768\n"						\
-		EX_TABLE_UA_FAULT(0b, 4b, %[rc])				\
-		EX_TABLE_UA_FAULT(1b, 4b, %[rc])				\
-		EX_TABLE_UA_FAULT(2b, 4b, %[rc])				\
-		EX_TABLE_UA_FAULT(3b, 4b, %[rc])				\
+		"3:	sacf	768\n"						\
+		EX_TABLE_INSN_UA_FAULT(0b, 3b, %[rc])				\
+		EX_TABLE_INSN_UA_FAULT(2b, 3b, %[rc])				\
 		: [rc] "=d" (rc), [old] "=&d" (*old),				\
 		  [new] "=&d" (new), [uaddr] "+Q" (*uaddr)			\
 		: [oparg] "d" (oparg)						\
@@ -86,10 +84,9 @@ int futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr, u32 oldval, u32
 	asm_inline volatile(
 		"	sacf	256\n"
 		"0:	cs	%[old],%[new],%[uaddr]\n"
-		"1:	lhi	%[rc],0\n"
-		"2:	sacf	768\n"
-		EX_TABLE_UA_FAULT(0b, 2b, %[rc])
-		EX_TABLE_UA_FAULT(1b, 2b, %[rc])
+		"	lhi	%[rc],0\n"
+		"1:	sacf	768\n"
+		EX_TABLE_INSN_UA_FAULT(0b, 1b, %[rc])
 		: [rc] "=d" (rc), [old] "+d" (oldval), [uaddr] "+Q" (*uaddr)
 		: [new] "d" (newval)
 		: "cc", "memory");
-- 
2.53.0

[PATCH 14/32] s390/diag: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all diag exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so duplicate entries and nopr padding are no longer needed and removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/diag.h    |  5 ++---
 arch/s390/include/asm/diag288.h |  6 +++---
 arch/s390/kernel/diag/diag.c    | 12 ++++++------
 3 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/arch/s390/include/asm/diag.h b/arch/s390/include/asm/diag.h
index 8db8db3b1018..1416ee1ad323 100644
--- a/arch/s390/include/asm/diag.h
+++ b/arch/s390/include/asm/diag.h
@@ -68,9 +68,8 @@ static inline void diag10_range(unsigned long start_pfn, unsigned long num_pfn)
 	diag_stat_inc(DIAG_STAT_X010);
 	asm_inline volatile(
 		"0:	diag	%0,%1,0x10\n"
-		"1:	nopr	%%r7\n"
-		EX_TABLE(0b, 1b)
-		EX_TABLE(1b, 1b)
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: : "a" (start_addr), "a" (end_addr));
 }
 
diff --git a/arch/s390/include/asm/diag288.h b/arch/s390/include/asm/diag288.h
index 5e1b43cea9d6..f0a37a80d672 100644
--- a/arch/s390/include/asm/diag288.h
+++ b/arch/s390/include/asm/diag288.h
@@ -28,10 +28,10 @@ static inline int __diag288(unsigned int func, unsigned int timeout,
 	int rc = -EINVAL;
 
 	asm volatile(
-		"	diag	%[r1],%[r3],0x288\n"
-		"0:	lhi	%[rc],0\n"
+		"0:	diag	%[r1],%[r3],0x288\n"
+		"	lhi	%[rc],0\n"
 		"1:"
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: [rc] "+d" (rc)
 		: [r1] "d" (r1.pair), [r3] "d" (r3.pair)
 		: "cc", "memory");
diff --git a/arch/s390/kernel/diag/diag.c b/arch/s390/kernel/diag/diag.c
index 56b862ba9be8..f23b3e2b070e 100644
--- a/arch/s390/kernel/diag/diag.c
+++ b/arch/s390/kernel/diag/diag.c
@@ -196,9 +196,9 @@ static inline int __diag204(unsigned long *subcode, unsigned long size, void *ad
 	union register_pair rp = { .even = *subcode, .odd = size };
 
 	asm_inline volatile(
-		"	diag	%[addr],%[rp],0x204\n"
-		"0:	nopr	%%r7\n"
-		EX_TABLE(0b,0b)
+		"0:	diag	%[addr],%[rp],0x204\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: [rp] "+&d" (rp.pair) : [addr] "d" (addr) : "memory");
 	*subcode = rp.even;
 	return rp.odd;
@@ -287,10 +287,10 @@ int diag224(void *ptr)
 
 	diag_stat_inc(DIAG_STAT_X224);
 	asm_inline volatile("\n"
-		"	diag	%[type],%[addr],0x224\n"
-		"0:	lhi	%[rc],0\n"
+		"0:	diag	%[type],%[addr],0x224\n"
+		"	lhi	%[rc],0\n"
 		"1:\n"
-		EX_TABLE(0b,1b)
+		EX_TABLE_INSN(0b, 1b)
 		: [rc] "+d" (rc)
 		, "=m" (*(struct { char buf[PAGE_SIZE]; } *)ptr)
 		: [type] "d" (0), [addr] "d" (addr));
-- 
2.53.0

[PATCH 15/32] s390/mm/pfault: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all pfault exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so nopr padding is no longer needed and removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/mm/pfault.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/s390/mm/pfault.c b/arch/s390/mm/pfault.c
index 6ecd6b0a22a8..8f37741471e2 100644
--- a/arch/s390/mm/pfault.c
+++ b/arch/s390/mm/pfault.c
@@ -58,9 +58,9 @@ int __pfault_init(void)
 		return rc;
 	diag_stat_inc(DIAG_STAT_X258);
 	asm_inline volatile(
-		"	diag	%[refbk],%[rc],0x258\n"
-		"0:	nopr	%%r7\n"
-		EX_TABLE(0b, 0b)
+		"0:	diag	%[refbk],%[rc],0x258\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: [rc] "+d" (rc)
 		: [refbk] "a" (virt_to_phys(&pfault_init_refbk)), "m" (pfault_init_refbk)
 		: "cc");
@@ -80,9 +80,9 @@ void __pfault_fini(void)
 		return;
 	diag_stat_inc(DIAG_STAT_X258);
 	asm_inline volatile(
-		"	diag	%[refbk],0,0x258\n"
-		"0:	nopr	%%r7\n"
-		EX_TABLE(0b, 0b)
+		"0:	diag	%[refbk],0,0x258\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		:
 		: [refbk] "a" (virt_to_phys(&pfault_fini_refbk)), "m" (pfault_fini_refbk)
 		: "cc");
-- 
2.53.0

[PATCH 16/32] s390/cpu_mf: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all cpu_mf exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so duplicate entries are removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/cpu_mf.h | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/arch/s390/include/asm/cpu_mf.h b/arch/s390/include/asm/cpu_mf.h
index 1798fbd59068..dd2991622e82 100644
--- a/arch/s390/include/asm/cpu_mf.h
+++ b/arch/s390/include/asm/cpu_mf.h
@@ -173,9 +173,9 @@ static inline int qctri(struct cpumf_ctr_info *info)
 
 	asm_inline volatile (
 		"0:	qctri	%1\n"
-		"1:	lhi	%0,0\n"
-		"2:\n"
-		EX_TABLE(1b, 2b)
+		"	lhi	%0,0\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: "+d" (rc), "=Q" (*info));
 	return rc;
 }
@@ -257,9 +257,9 @@ static inline int qsi(struct hws_qsi_info_block *info)
 
 	asm volatile(
 		"0:	qsi	%1\n"
-		"1:	lhi	%0,0\n"
-		"2:\n"
-		EX_TABLE(0b, 2b) EX_TABLE(1b, 2b)
+		"	lhi	%0,0\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: "+d" (cc), "+Q" (*info));
 	return cc ? -EINVAL : 0;
 }
@@ -272,10 +272,10 @@ static inline int lsctl(struct hws_lsctl_request_block *req)
 	exception = 1;
 	asm volatile(
 		"0:	lsctl	%[req]\n"
-		"1:	lhi	%[exc],0\n"
-		"2:\n"
+		"	lhi	%[exc],0\n"
+		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 2b) EX_TABLE(1b, 2b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [exc] "+d" (exception)
 		: [req] "Q" (*req)
 		: CC_CLOBBER);
-- 
2.53.0

[PATCH 17/32] s390/cert_store: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the diag320 exception table entry to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so nopr padding is no longer needed and removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/cert_store.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/s390/kernel/cert_store.c b/arch/s390/kernel/cert_store.c
index dc1992a675de..998f2da56175 100644
--- a/arch/s390/kernel/cert_store.c
+++ b/arch/s390/kernel/cert_store.c
@@ -236,9 +236,9 @@ static int __diag320(unsigned long subcode, void *addr)
 	union register_pair rp = { .even = (unsigned long)addr, };
 
 	asm_inline volatile(
-		"	diag	%[rp],%[subcode],0x320\n"
-		"0:	nopr	%%r7\n"
-		EX_TABLE(0b, 0b)
+		"0:	diag	%[rp],%[subcode],0x320\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: [rp] "+d" (rp.pair)
 		: [subcode] "d" (subcode)
 		: "cc", "memory");
-- 
2.53.0

[PATCH 18/32] s390/ipl: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the diag308 exception table entry to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so nopr padding is no longer needed and removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/ipl.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/s390/kernel/ipl.c b/arch/s390/kernel/ipl.c
index b1e798f8e1dd..c441b1357492 100644
--- a/arch/s390/kernel/ipl.c
+++ b/arch/s390/kernel/ipl.c
@@ -188,9 +188,9 @@ static inline int __diag308(unsigned long subcode, unsigned long addr)
 	r1.even = addr;
 	r1.odd	= 0;
 	asm_inline volatile(
-		"	diag	%[r1],%[subcode],0x308\n"
-		"0:	nopr	%%r7\n"
-		EX_TABLE(0b,0b)
+		"0:	diag	%[r1],%[subcode],0x308\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: [r1] "+&d" (r1.pair)
 		: [subcode] "d" (subcode)
 		: "cc", "memory");
-- 
2.53.0

[PATCH 19/32] s390/hypfs: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the diag2fc exception table entry to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so nopr padding is no longer needed and removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/hypfs/hypfs_vm.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/s390/hypfs/hypfs_vm.c b/arch/s390/hypfs/hypfs_vm.c
index 4db2895e4da3..bd47fce59731 100644
--- a/arch/s390/hypfs/hypfs_vm.c
+++ b/arch/s390/hypfs/hypfs_vm.c
@@ -42,9 +42,9 @@ static int diag2fc(int size, char* query, void *addr)
 
 	diag_stat_inc(DIAG_STAT_X2FC);
 	asm volatile(
-		"	diag    %0,%1,0x2fc\n"
-		"0:	nopr	%%r7\n"
-		EX_TABLE(0b,0b)
+		"0:	diag    %0,%1,0x2fc\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: "=d" (residual_cnt), "+d" (rc) : "0" (&parm_list) : "memory");
 
 	if ((rc != 0 ) && (rc != -2))
-- 
2.53.0

[PATCH 20/32] s390/ap: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all ap exception table entries to use EX_TABLE_INSN.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/ap.h | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/s390/include/asm/ap.h b/arch/s390/include/asm/ap.h
index c91b6ace199d..b0f8e3946a43 100644
--- a/arch/s390/include/asm/ap.h
+++ b/arch/s390/include/asm/ap.h
@@ -91,10 +91,10 @@ static inline bool ap_instructions_available(void)
 		"	lgr	0,%[reg0]\n"		/* qid into gr0 */
 		"	lghi	1,0\n"			/* 0 into gr1 */
 		"	lghi	2,0\n"			/* 0 into gr2 */
-		"	.insn	rre,0xb2af0000,0,0\n"	/* PQAP(TAPQ) */
-		"0:	la	%[reg1],1\n"		/* 1 into reg1 */
+		"0:	.insn	rre,0xb2af0000,0,0\n"	/* PQAP(TAPQ) */
+		"	la	%[reg1],1\n"		/* 1 into reg1 */
 		"1:\n"
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: [reg1] "+&d" (reg1)
 		: [reg0] "d" (reg0)
 		: "cc", "0", "1", "2");
@@ -272,10 +272,10 @@ static inline int ap_qci(struct ap_config_info *config)
 	asm volatile(
 		"	lgr	0,%[reg0]\n"		/* QCI fc into gr0 */
 		"	lgr	2,%[reg2]\n"		/* ptr to config into gr2 */
-		"	.insn	rre,0xb2af0000,0,0\n"	/* PQAP(QCI) */
-		"0:	la	%[reg1],0\n"		/* good case, QCI fc available */
+		"0:	.insn	rre,0xb2af0000,0,0\n"	/* PQAP(QCI) */
+		"	la	%[reg1],0\n"		/* good case, QCI fc available */
 		"1:\n"
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: [reg1] "+&d" (reg1)
 		: [reg0] "d" (reg0), [reg2] "d" (reg2)
 		: "cc", "memory", "0", "2");
-- 
2.53.0

[PATCH 21/32] s390/fpu: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the fpu_lfpc_safe exception table entry to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so nopr padding is no longer needed and removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/fpu-insn.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/s390/include/asm/fpu-insn.h b/arch/s390/include/asm/fpu-insn.h
index ae8b7033cfd2..eeb0a233d5c7 100644
--- a/arch/s390/include/asm/fpu-insn.h
+++ b/arch/s390/include/asm/fpu-insn.h
@@ -104,9 +104,9 @@ static inline void fpu_lfpc_safe(unsigned int *fpc)
 {
 	instrument_read(fpc, sizeof(*fpc));
 	asm_inline volatile(
-		"	lfpc	%[fpc]\n"
-		"0:	nopr	%%r7\n"
-		EX_TABLE_FPC(0b, 0b)
+		"0:	lfpc	%[fpc]\n"
+		"1:\n"
+		EX_TABLE_INSN_FPC(0b, 1b)
 		:
 		: [fpc] "Q" (*fpc)
 		: "memory");
-- 
2.53.0

[PATCH 22/32] s390/traps: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the monitor call exception table entry to use EX_TABLE_INSN.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/traps.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/s390/kernel/traps.c b/arch/s390/kernel/traps.c
index a0937b4a6cf3..7d45fa6a21dc 100644
--- a/arch/s390/kernel/traps.c
+++ b/arch/s390/kernel/traps.c
@@ -312,10 +312,10 @@ static void __init test_monitor_call(void)
 	if (!IS_ENABLED(CONFIG_BUG))
 		return;
 	asm_inline volatile(
-		"	mc	%[monc](%%r0),0\n"
-		"0:	lhi	%[val],0\n"
+		"0:	mc	%[monc](%%r0),0\n"
+		"	lhi	%[val],0\n"
 		"1:\n"
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: [val] "+d" (val)
 		: [monc] "i" (MONCODE_BUG));
 	if (!val)
-- 
2.53.0

[PATCH 23/32] s390/word-at-a-time: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the load_unaligned_zeropad exception table entries to use
EX_TABLE_INSN. With EX_TABLE_INSN entries always point to the faulting
instruction, so the duplicate entry and nopr padding are no longer
needed and removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/include/asm/word-at-a-time.h | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/arch/s390/include/asm/word-at-a-time.h b/arch/s390/include/asm/word-at-a-time.h
index eaa19dee7699..0b7779c8d5a4 100644
--- a/arch/s390/include/asm/word-at-a-time.h
+++ b/arch/s390/include/asm/word-at-a-time.h
@@ -54,9 +54,8 @@ static inline unsigned long load_unaligned_zeropad(const void *addr)
 
 	asm_inline volatile(
 		"0:	lg	%[data],0(%[addr])\n"
-		"1:	nopr	%%r7\n"
-		EX_TABLE_ZEROPAD(0b, 1b, %[data], %[addr])
-		EX_TABLE_ZEROPAD(1b, 1b, %[data], %[addr])
+		"1:\n"
+		EX_TABLE_INSN_ZEROPAD(0b, 1b, %[data], %[addr])
 		: [data] "=d" (data)
 		: [addr] "a" (addr), "m" (*(unsigned long *)addr));
 	return data;
-- 
2.53.0

[PATCH 24/32] s390/kvm: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the gaccess mvcos exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so duplicate entries are removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kvm/s390/gaccess.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/arch/s390/kvm/s390/gaccess.c b/arch/s390/kvm/s390/gaccess.c
index 405345ccc4f1..2d27dcb68002 100644
--- a/arch/s390/kvm/s390/gaccess.c
+++ b/arch/s390/kvm/s390/gaccess.c
@@ -838,12 +838,11 @@ static int mvcos_key(void *to, const void *from, unsigned long size, u8 dst_key,
 	int exception = PGM_PROTECTION;
 
 	asm_inline volatile(
-		"       lr      %%r0,%[spec]\n"
-		"0:     mvcos   %[to],%[from],%[size]\n"
-		"1:     lhi     %[exc],0\n"
-		"2:\n"
-		EX_TABLE(0b, 2b)
-		EX_TABLE(1b, 2b)
+		"	lr	%%r0,%[spec]\n"
+		"0:	mvcos	%[to],%[from],%[size]\n"
+		"	lhi	%[exc],0\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: [size] "+d" (size), [to] "=Q" (*(char *)to), [exc] "+d" (exception)
 		: [spec] "d" (spec.val), [from] "Q" (*(const char *)from)
 		: "memory", "cc", "0");
-- 
2.53.0

[PATCH 25/32] s390/pci: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all pci exception table entries to use EX_TABLE_INSN.
With EX_TABLE_INSN entries always point to the faulting instruction,
so duplicate entries are removed.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/pci/pci_clp.c  | 12 ++++++------
 arch/s390/pci/pci_insn.c | 36 ++++++++++++++++++------------------
 arch/s390/pci/pci_mmio.c | 28 +++++++++++++++-------------
 3 files changed, 39 insertions(+), 37 deletions(-)

diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c
index 177aa0214547..2d7eaf4d9d59 100644
--- a/arch/s390/pci/pci_clp.c
+++ b/arch/s390/pci/pci_clp.c
@@ -55,11 +55,11 @@ static inline int clp_get_ilp(unsigned long *ilp)
 
 	exception = 1;
 	asm_inline volatile (
-		"	.insn	rrf,0xb9a00000,%[mask],%[cmd],8,0\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rrf,0xb9a00000,%[mask],%[cmd],8,0\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [mask] "=d" (mask), [exc] "+d" (exception)
 		: [cmd] "a" (1)
 		: CC_CLOBBER);
@@ -78,11 +78,11 @@ static __always_inline int clp_req(void *data, unsigned int lps)
 
 	exception = 1;
 	asm_inline volatile (
-		"	.insn	rrf,0xb9a00000,%[ign],%[req],0,%[lps]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rrf,0xb9a00000,%[ign],%[req],0,%[lps]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [ign] "=d" (ignored), "+m" (*req), [exc] "+d" (exception)
 		: [req] "a" (req), [lps] "i" (lps)
 		: CC_CLOBBER);
diff --git a/arch/s390/pci/pci_insn.c b/arch/s390/pci/pci_insn.c
index 35ceb1bea1c6..ed0def408f18 100644
--- a/arch/s390/pci/pci_insn.c
+++ b/arch/s390/pci/pci_insn.c
@@ -161,11 +161,11 @@ static inline int ____pcilg(u64 *data, u64 req, u64 offset, u8 *status)
 
 	exception = 1;
 	asm_inline volatile (
-		"	.insn	rre,0xb9d20000,%[data],%[req_off]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rre,0xb9d20000,%[data],%[req_off]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [data] "=d" (__data),
 		  [req_off] "+d" (req_off.pair), [exc] "+d" (exception)
 		:
@@ -230,11 +230,11 @@ static inline int __pcilg_mio(u64 *data, u64 ioaddr, u64 len, u8 *status)
 
 	exception = 1;
 	asm_inline volatile (
-		"       .insn   rre,0xb9d60000,%[data],%[ioaddr_len]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rre,0xb9d60000,%[data],%[ioaddr_len]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [data] "=d" (__data),
 		  [ioaddr_len] "+d" (ioaddr_len.pair), [exc] "+d" (exception)
 		:
@@ -268,11 +268,11 @@ static inline int __pcistg(u64 data, u64 req, u64 offset, u8 *status)
 
 	exception = 1;
 	asm_inline volatile (
-		"	.insn	rre,0xb9d00000,%[data],%[req_off]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rre,0xb9d00000,%[data],%[req_off]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [req_off] "+d" (req_off.pair), [exc] "+d" (exception)
 		: [data] "d" (data)
 		: CC_CLOBBER);
@@ -322,11 +322,11 @@ static inline int __pcistg_mio(u64 data, u64 ioaddr, u64 len, u8 *status)
 
 	exception = 1;
 	asm_inline volatile (
-		"       .insn   rre,0xb9d40000,%[data],%[ioaddr_len]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rre,0xb9d40000,%[data],%[ioaddr_len]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [ioaddr_len] "+d" (ioaddr_len.pair), [exc] "+d" (exception)
 		: [data] "d" (data)
 		: CC_CLOBBER_LIST("memory"));
@@ -357,11 +357,11 @@ static inline int __pcistb(const u64 *data, u64 req, u64 offset, u8 *status)
 
 	exception = 1;
 	asm_inline volatile (
-		"	.insn	rsy,0xeb00000000d0,%[req],%[offset],%[data]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rsy,0xeb00000000d0,%[req],%[offset],%[data]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [req] "+d" (req), [exc] "+d" (exception)
 		: [offset] "d" (offset), [data] "Q" (*data)
 		: CC_CLOBBER);
@@ -411,11 +411,11 @@ static inline int __pcistb_mio(const u64 *data, u64 ioaddr, u64 len, u8 *status)
 
 	exception = 1;
 	asm_inline volatile (
-		"       .insn   rsy,0xeb00000000d4,%[len],%[ioaddr],%[data]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rsy,0xeb00000000d4,%[len],%[ioaddr],%[data]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [len] "+d" (len), [exc] "+d" (exception)
 		: [ioaddr] "d" (ioaddr), [data] "Q" (*data)
 		: CC_CLOBBER);
diff --git a/arch/s390/pci/pci_mmio.c b/arch/s390/pci/pci_mmio.c
index f3f79ba78410..8b49175b0031 100644
--- a/arch/s390/pci/pci_mmio.c
+++ b/arch/s390/pci/pci_mmio.c
@@ -39,10 +39,10 @@ static inline int __pcistb_mio_inuser(
 	asm_inline volatile (
 		"	sacf	256\n"
 		"0:	.insn	rsy,0xeb00000000d4,%[len],%[ioaddr],%[src]\n"
-		"1:	lhi	%[exc],0\n"
-		"2:	sacf	768\n"
+		"	lhi	%[exc],0\n"
+		"1:	sacf	768\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 2b) EX_TABLE(1b, 2b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [len] "+d" (len), [exc] "+d" (exception)
 		: [ioaddr] "a" (ioaddr), [src] "Q" (*((u8 __force *)src))
 		: CC_CLOBBER_LIST("memory"));
@@ -72,15 +72,16 @@ static inline int __pcistg_mio_inuser(
 	asm_inline volatile (
 		"	sacf	256\n"
 		"0:	llgc	%[tmp],0(%[src])\n"
-		"4:	sllg	%[val],%[val],8\n"
+		"	sllg	%[val],%[val],8\n"
 		"	aghi	%[src],1\n"
 		"	ogr	%[val],%[tmp]\n"
 		"	brctg	%[cnt],0b\n"
 		"1:	.insn	rre,0xb9d40000,%[val],%[ioaddr_len]\n"
-		"2:	lhi	%[exc],0\n"
-		"3:	sacf	768\n"
+		"	lhi	%[exc],0\n"
+		"2:	sacf	768\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 3b) EX_TABLE(4b, 3b) EX_TABLE(1b, 3b) EX_TABLE(2b, 3b)
+		EX_TABLE_INSN(0b, 2b)
+		EX_TABLE_INSN(1b, 2b)
 		: [src] "+a" (src), [cnt] "+d" (cnt),
 		  [val] "+d" (val), [tmp] "=d" (tmp), [exc] "+d" (exception),
 		  CC_OUT(cc, cc), [ioaddr_len] "+&d" (ioaddr_len.pair)
@@ -227,13 +228,13 @@ static inline int __pcilg_mio_inuser(
 	asm_inline volatile (
 		"	sacf	256\n"
 		"0:	.insn	rre,0xb9d60000,%[val],%[ioaddr_len]\n"
-		"1:	lhi	%[exc],0\n"
+		"	lhi	%[exc],0\n"
 		"	jne	4f\n"
-		"2:	ahi	%[shift],-8\n"
+		"1:	ahi	%[shift],-8\n"
 		"	srlg	%[tmp],%[val],0(%[shift])\n"
-		"3:	stc	%[tmp],0(%[dst])\n"
-		"5:	aghi	%[dst],1\n"
-		"	brctg	%[cnt],2b\n"
+		"2:	stc	%[tmp],0(%[dst])\n"
+		"	aghi	%[dst],1\n"
+		"	brctg	%[cnt],1b\n"
 		/*
 		 * Use xr to clear exc and set condition code to zero
 		 * to ensure flag output is correct for this branch.
@@ -241,7 +242,8 @@ static inline int __pcilg_mio_inuser(
 		"	xr	%[exc],%[exc]\n"
 		"4:	sacf	768\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 4b) EX_TABLE(1b, 4b) EX_TABLE(3b, 4b) EX_TABLE(5b, 4b)
+		EX_TABLE_INSN(0b, 4b)
+		EX_TABLE_INSN(2b, 4b)
 		: [ioaddr_len] "+&d" (ioaddr_len.pair), [exc] "+d" (exception),
 		  CC_OUT(cc, cc), [val] "=d" (val),
 		  [dst] "+a" (dst), [cnt] "+d" (cnt), [tmp] "=d" (tmp),
-- 
2.53.0

[PATCH 26/32] s390/boot: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all boot exception table entries to use EX_TABLE_INSN.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/boot/ipl_parm.c     |  6 +++---
 arch/s390/boot/physmem_info.c | 18 +++++++++---------
 arch/s390/boot/startup.c      | 12 ++++++------
 3 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/arch/s390/boot/ipl_parm.c b/arch/s390/boot/ipl_parm.c
index c1b43e5e688a..ff2ac7f16fe0 100644
--- a/arch/s390/boot/ipl_parm.c
+++ b/arch/s390/boot/ipl_parm.c
@@ -40,9 +40,9 @@ static inline int __diag308(unsigned long subcode, void *addr)
 	union register_pair r1 = { .even = (unsigned long)addr, .odd = 0 };
 
 	asm_inline volatile(
-		"	diag	%[r1],%[subcode],0x308\n"
-		"0:\n"
-		EX_TABLE(0b, 0b)
+		"0:	diag	%[r1],%[subcode],0x308\n"
+		"1:\n"
+		EX_TABLE_INSN(0b, 1b)
 		: [r1] "+d" (r1.pair)
 		: [subcode] "d" (subcode)
 		: "cc", "memory");
diff --git a/arch/s390/boot/physmem_info.c b/arch/s390/boot/physmem_info.c
index 0ebb2174713f..e5248b637326 100644
--- a/arch/s390/boot/physmem_info.c
+++ b/arch/s390/boot/physmem_info.c
@@ -68,11 +68,11 @@ static int __diag260(unsigned long rx1, unsigned long rx2)
 	ry = 0x10; /* storage configuration */
 	exception = 1;
 	asm_inline volatile(
-		"	diag	%[rx],%[ry],0x260\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	diag	%[rx],%[ry],0x260\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [exc] "+d" (exception), [ry] "+d" (ry)
 		: [rx] "d" (rx.pair)
 		: CC_CLOBBER_LIST("memory"));
@@ -108,9 +108,9 @@ static int diag500_storage_limit(unsigned long *max_physmem_end)
 	asm_inline volatile(
 		"	lghi	%%r1,%[subcode]\n"
 		"	lghi	%%r2,0\n"
-		"	diag	%%r2,%%r4,0x500\n"
-		"0:	lgr	%[slimit],%%r2\n"
-		EX_TABLE(0b, 0b)
+		"0:	diag	%%r2,%%r4,0x500\n"
+		"1:	lgr	%[slimit],%%r2\n"
+		EX_TABLE_INSN(0b, 1b)
 		: [slimit] "=d" (storage_limit)
 		: [subcode] "i" (DIAG500_SC_STOR_LIMIT)
 		: "memory", "1", "2");
@@ -127,11 +127,11 @@ static int tprot(unsigned long addr)
 
 	exception = 1;
 	asm_inline volatile(
-		"	tprot	0(%[addr]),0\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	tprot	0(%[addr]),0\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [exc] "+d" (exception)
 		: [addr] "a" (addr)
 		: CC_CLOBBER_LIST("memory"));
diff --git a/arch/s390/boot/startup.c b/arch/s390/boot/startup.c
index c1dcf8b1b579..5eb7a3e1f9f1 100644
--- a/arch/s390/boot/startup.c
+++ b/arch/s390/boot/startup.c
@@ -89,10 +89,10 @@ static void detect_diag9c(void)
 
 	cpu = stap();
 	asm_inline volatile(
-		"	diag	%[cpu],%%r0,0x9c\n"
-		"0:	lhi	%[rc],0\n"
+		"0:	diag	%[cpu],%%r0,0x9c\n"
+		"	lhi	%[rc],0\n"
 		"1:\n"
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: [rc] "+d" (rc)
 		: [cpu] "d" (cpu)
 		: "cc", "memory");
@@ -150,10 +150,10 @@ static int cmma_test_essa(void)
 
 	/* Test ESSA_GET_STATE */
 	asm_inline volatile(
-		"	.insn	rrf,0xb9ab0000,%[tmp],%[tmp],%[cmd],0\n"
-		"0:	lhi	%[rc],0\n"
+		"0:	.insn	rrf,0xb9ab0000,%[tmp],%[tmp],%[cmd],0\n"
+		"	lhi	%[rc],0\n"
 		"1:\n"
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: [rc] "+d" (rc), [tmp] "+d" (tmp)
 		: [cmd] "i" (ESSA_GET_STATE)
 		: "cc", "memory");
-- 
2.53.0

[PATCH 27/32] s390/cio: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert all cio exception table entries to use EX_TABLE_INSN.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 drivers/s390/cio/ioasm.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/drivers/s390/cio/ioasm.c b/drivers/s390/cio/ioasm.c
index 8b06b234e110..4974e80b0568 100644
--- a/drivers/s390/cio/ioasm.c
+++ b/drivers/s390/cio/ioasm.c
@@ -24,11 +24,11 @@ static inline int __stsch(struct subchannel_id schid, struct schib *addr)
 	exception = 1;
 	asm_inline volatile(
 		"	lgr	1,%[r1]\n"
-		"	stsch	%[addr]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	stsch	%[addr]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, ccode), [addr] "=Q" (*addr), [exc] "+d" (exception)
 		: [r1] "d" (r1)
 		: CC_CLOBBER_LIST("1"));
@@ -54,11 +54,11 @@ static inline int __msch(struct subchannel_id schid, struct schib *addr)
 	exception = 1;
 	asm_inline volatile(
 		"	lgr	1,%[r1]\n"
-		"	msch	%[addr]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	msch	%[addr]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, ccode), [exc] "+d" (exception)
 		: [r1] "d" (r1), [addr] "Q" (*addr)
 		: CC_CLOBBER_LIST("1"));
@@ -108,11 +108,11 @@ static inline int __ssch(struct subchannel_id schid, union orb *addr)
 	exception = 1;
 	asm_inline volatile(
 		"	lgr	1,%[r1]\n"
-		"	ssch	%[addr]\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	ssch	%[addr]\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, ccode), [exc] "+d" (exception)
 		: [r1] "d" (r1), [addr] "Q" (*addr)
 		: CC_CLOBBER_LIST("memory", "1"));
@@ -179,11 +179,11 @@ int chsc(void *chsc_area)
 
 	exception = 1;
 	asm_inline volatile(
-		"	.insn	rre,0xb25f0000,%[chsc_area],0\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	.insn	rre,0xb25f0000,%[chsc_area],0\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), "+m" (*(addr_type *)chsc_area), [exc] "+d" (exception)
 		: [chsc_area] "d" (chsc_area)
 		: CC_CLOBBER);
-- 
2.53.0

[PATCH 28/32] s390/dasd: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the dasd_diag exception table entry to use EX_TABLE_INSN.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 drivers/s390/block/dasd_diag.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/s390/block/dasd_diag.c b/drivers/s390/block/dasd_diag.c
index af68c2ee659d..ce391d35adf9 100644
--- a/drivers/s390/block/dasd_diag.c
+++ b/drivers/s390/block/dasd_diag.c
@@ -77,11 +77,11 @@ static inline int __dia250(void *iob, int cmd)
 
 	exception = 1;
 	asm_inline volatile(
-		"	diag	%[rx],%[cmd],0x250\n"
-		"0:	lhi	%[exc],0\n"
+		"0:	diag	%[rx],%[cmd],0x250\n"
+		"	lhi	%[exc],0\n"
 		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b,1b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [rx] "+d" (rx.pair),
 		  "+m" (*(addr_type *)iob), [exc] "+d" (exception)
 		: [cmd] "d" (cmd)
-- 
2.53.0

[PATCH 29/32] s390/diag_ftp: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the diag_ftp exception table entry to use EX_TABLE_INSN.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 drivers/s390/char/diag_ftp.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/s390/char/diag_ftp.c b/drivers/s390/char/diag_ftp.c
index a1e110c96f74..0a506559799e 100644
--- a/drivers/s390/char/diag_ftp.c
+++ b/drivers/s390/char/diag_ftp.c
@@ -106,11 +106,11 @@ static int diag_ftp_2c4(struct diag_ftp_ldfpl *fpl,
 
 	diag_stat_inc(DIAG_STAT_X2C4);
 	asm_inline volatile(
-		"	diag	%[addr],%[cmd],0x2c4\n"
-		"0:	j	2f\n"
+		"0:	diag	%[addr],%[cmd],0x2c4\n"
+		"	j	2f\n"
 		"1:	la	%[rc],%[err]\n"
 		"2:\n"
-		EX_TABLE(0b, 1b)
+		EX_TABLE_INSN(0b, 1b)
 		: [rc] "=d" (rc), "+m" (*fpl)
 		: [cmd] "0" (cmd), [addr] "d" (virt_to_phys(fpl)),
 		  [err] "i" (DIAG_FTP_RET_EPERM)
-- 
2.53.0

[PATCH 30/32] s390/sclp: Convert inline assembly to EX_TABLE_INSN

[Heiko Carstens]

Convert the sclp_service_call exception table entries to use
EX_TABLE_INSN.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 drivers/s390/char/sclp.h | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/s390/char/sclp.h b/drivers/s390/char/sclp.h
index b31a680e0871..b91d31f8e0ef 100644
--- a/drivers/s390/char/sclp.h
+++ b/drivers/s390/char/sclp.h
@@ -320,11 +320,10 @@ static inline int sclp_service_call(sclp_cmdw_t command, void *sccb)
 	exception = 1;
 	asm_inline volatile(
 		"0:	.insn	rre,0xb2200000,%[cmd],%[sccb]\n" /* servc */
-		"1:	lhi	%[exc],0\n"
-		"2:\n"
+		"	lhi	%[exc],0\n"
+		"1:\n"
 		CC_IPM(cc)
-		EX_TABLE(0b, 2b)
-		EX_TABLE(1b, 2b)
+		EX_TABLE_INSN(0b, 1b)
 		: CC_OUT(cc, cc), [exc] "+d" (exception)
 		: [cmd] "d" (command), [sccb] "a" (__pa(sccb))
 		: CC_CLOBBER_LIST("memory"));
-- 
2.53.0

[PATCH 31/32] s390/bpf: Simplify exception table entry handling

[Heiko Carstens]

With exception table lookup always pointing to the faulting instruction,
only a single exception table entry per probe instruction is needed.

Remove the second exception table entry, the extra nop instruction
emitted between/after probe instructions, and simplify probe handling
in the BPF JIT.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/net/bpf_jit_comp.c | 78 ++++++++++--------------------------
 1 file changed, 21 insertions(+), 57 deletions(-)

diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
index c4b47070bb59..9f1632e068e1 100644
--- a/arch/s390/net/bpf_jit_comp.c
+++ b/arch/s390/net/bpf_jit_comp.c
@@ -723,7 +723,6 @@ bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)
  */
 struct bpf_jit_probe {
 	int prg;	/* JITed instruction offset */
-	int nop_prg;	/* JITed nop offset */
 	int reg;	/* Register to clear on exception */
 	int arena_reg;	/* Register to use for arena addressing */
 };
@@ -731,29 +730,10 @@ struct bpf_jit_probe {
 static void bpf_jit_probe_init(struct bpf_jit_probe *probe)
 {
 	probe->prg = -1;
-	probe->nop_prg = -1;
 	probe->reg = -1;
 	probe->arena_reg = REG_0;
 }
 
-/*
- * Handlers of certain exceptions leave psw.addr pointing to the instruction
- * directly after the failing one. Therefore, create two exception table
- * entries and also add a nop in case two probing instructions come directly
- * after each other.
- */
-static void bpf_jit_probe_emit_nop(struct bpf_jit *jit,
-				   struct bpf_jit_probe *probe)
-{
-	if (probe->prg == -1 || probe->nop_prg != -1)
-		/* The probe is not armed or nop is already emitted. */
-		return;
-
-	probe->nop_prg = jit->prg;
-	/* bcr 0,%0 */
-	_EMIT2(0x0700);
-}
-
 static void bpf_jit_probe_load_pre(struct bpf_jit *jit, struct bpf_insn *insn,
 				   struct bpf_jit_probe *probe)
 {
@@ -815,47 +795,36 @@ static int bpf_jit_probe_post(struct bpf_jit *jit, struct bpf_prog *fp,
 			      struct bpf_jit_probe *probe)
 {
 	struct exception_table_entry *ex;
-	int i, prg;
 	s64 delta;
-	u8 *insn;
 
 	if (probe->prg == -1)
 		/* The probe is not armed. */
 		return 0;
-	bpf_jit_probe_emit_nop(jit, probe);
 	if (!fp->aux->extable)
 		/* Do nothing during early JIT passes. */
 		return 0;
-	insn = jit->prg_buf + probe->prg;
-	if (WARN_ON_ONCE(probe->prg + insn_length(*insn) != probe->nop_prg))
-		/* JIT bug - gap between probe and nop instructions. */
+	if (WARN_ON_ONCE(jit->excnt >= fp->aux->num_exentries))
+		/* Verifier bug - not enough entries. */
 		return -1;
-	for (i = 0; i < 2; i++) {
-		if (WARN_ON_ONCE(jit->excnt >= fp->aux->num_exentries))
-			/* Verifier bug - not enough entries. */
-			return -1;
-		ex = &fp->aux->extable[jit->excnt];
-		/* Add extable entries for probe and nop instructions. */
-		prg = i == 0 ? probe->prg : probe->nop_prg;
-		delta = jit->prg_buf + prg - (u8 *)&ex->insn;
-		if (WARN_ON_ONCE(delta < INT_MIN || delta > INT_MAX))
-			/* JIT bug - code and extable must be close. */
-			return -1;
-		ex->insn = delta;
-		/*
-		 * Land on the current instruction. Note that the extable
-		 * infrastructure ignores the fixup field; it is handled by
-		 * ex_handler_bpf().
-		 */
-		delta = jit->prg_buf + jit->prg - (u8 *)&ex->fixup;
-		if (WARN_ON_ONCE(delta < INT_MIN || delta > INT_MAX))
-			/* JIT bug - landing pad and extable must be close. */
-			return -1;
-		ex->fixup = delta;
-		ex->type = EX_TYPE_BPF;
-		ex->data = probe->reg;
-		jit->excnt++;
-	}
+	ex = &fp->aux->extable[jit->excnt];
+	delta = jit->prg_buf + probe->prg - (u8 *)&ex->insn;
+	if (WARN_ON_ONCE(delta < INT_MIN || delta > INT_MAX))
+		/* JIT bug - code and extable must be close. */
+		return -1;
+	ex->insn = delta;
+	/*
+	 * Land on the current instruction. Note that the extable
+	 * infrastructure ignores the fixup field; it is handled by
+	 * ex_handler_bpf().
+	 */
+	delta = jit->prg_buf + jit->prg - (u8 *)&ex->fixup;
+	if (WARN_ON_ONCE(delta < INT_MIN || delta > INT_MAX))
+		/* JIT bug - landing pad and extable must be close. */
+		return -1;
+	ex->fixup = delta;
+	ex->type = EX_TYPE_BPF;
+	ex->data = probe->reg;
+	jit->excnt++;
 	return 0;
 }
 
@@ -1705,7 +1674,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
 			EMIT6_DISP_LH(0xe3000000,
 				      is32 ? 0x0058 : 0x0004, REG_W0, REG_0,
 				      load_probe.arena_reg, off);
-			bpf_jit_probe_emit_nop(jit, &load_probe);
 			/* Reuse {ly|lg}'s arena_reg for {csy|csg}. */
 			if (load_probe.prg != -1) {
 				probe.prg = jit->prg;
@@ -1716,7 +1684,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
 			/* 0: {csy|csg} %w0,%src,off(%arena) */
 			EMIT6_DISP_LH(0xeb000000, is32 ? 0x0014 : 0x0030,
 				      REG_W0, src_reg, probe.arena_reg, off);
-			bpf_jit_probe_emit_nop(jit, &probe);
 			/* brc 4,0b */
 			EMIT4_PCREL_RIC(0xa7040000, 4, loop_start);
 			/* {llgfr|lgr} %src,%w0 */
@@ -2364,9 +2331,6 @@ static struct bpf_binary_header *bpf_jit_alloc(struct bpf_jit *jit,
 			 */
 			fp->aux->num_exentries += 1;
 	}
-	/* We need two entries per insn. */
-	fp->aux->num_exentries *= 2;
-
 	code_size = roundup(jit->size,
 			    __alignof__(struct exception_table_entry));
 	extable_size = fp->aux->num_exentries *
-- 
2.53.0

[PATCH 32/32] s390/extable: Remove EX_TABLE macros and fallback extable search

[Heiko Carstens]

Now that all exception table entries point to the faulting instruction,
rename all EX_TABLE* macros to EX_TABLE_INSN* and remove the
EX_TABLE_INSN* aliases.

Also remove the fallback extable search via instruction_pointer() which
was only needed to handle the old EX_TABLE entries that pointed to the
instruction after the faulting instruction.

Add a comment above the EX_TABLE_INSN macros that documents the usage
rule and describe the difference to the previous EX_TABLE macros. This
might be helpful in case of backporting to old kernels.

Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/boot/pgm_check.c          | 23 +++++------------
 arch/s390/include/asm/asm-extable.h | 38 ++++++++++++++---------------
 arch/s390/include/asm/extable.h     | 25 ++++++-------------
 arch/s390/mm/extable.c              |  2 --
 4 files changed, 32 insertions(+), 56 deletions(-)

diff --git a/arch/s390/boot/pgm_check.c b/arch/s390/boot/pgm_check.c
index 56d0be7e2823..e5a454b21669 100644
--- a/arch/s390/boot/pgm_check.c
+++ b/arch/s390/boot/pgm_check.c
@@ -51,31 +51,20 @@ static unsigned long boot_get_trap_ip(struct pt_regs *regs)
 	return __rewind_psw(regs->psw, regs->int_code >> 16);
 }
 
-static const struct exception_table_entry *search_boot_extable(unsigned long ip)
+static bool ex_handler(struct pt_regs *regs)
 {
 	const struct exception_table_entry *ex;
+	unsigned long ip = boot_get_trap_ip(regs);
 
 	for (ex = __start___ex_table; ex < __stop___ex_table; ex++) {
 		if (extable_insn(ex) != ip)
 			continue;
 		if (ex->type != EX_TYPE_FIXUP)
-			return NULL;
-		return ex;
+			return false;
+		regs->psw.addr = extable_fixup(ex);
+		return true;
 	}
-	return NULL;
-}
-
-static bool ex_handler(struct pt_regs *regs)
-{
-	const struct exception_table_entry *ex;
-
-	ex = search_boot_extable(boot_get_trap_ip(regs));
-	if (!ex)
-		ex = search_boot_extable(instruction_pointer(regs));
-	if (!ex)
-		return false;
-	regs->psw.addr = extable_fixup(ex);
-	return true;
+	return false;
 }
 
 void do_pgm_check(struct pt_regs *regs)
diff --git a/arch/s390/include/asm/asm-extable.h b/arch/s390/include/asm/asm-extable.h
index 488bd10452b3..fd43ecd3e491 100644
--- a/arch/s390/include/asm/asm-extable.h
+++ b/arch/s390/include/asm/asm-extable.h
@@ -65,41 +65,41 @@
 	stringify_in_c(.purgem	extable_reg;)					\
 	stringify_in_c(.previous)
 
-#define EX_TABLE(_fault, _target)					\
+/*
+ * EX_TABLE_INSN usage rules:
+ *
+ * _fault must point to the faulting instruction, never behind it.
+ *
+ * This differs from the former EX_TABLE convention, where usually two entries
+ * were needed. One that pointed to the faulting instruction (for nullifying
+ * exceptions), and a second one that pointed behind the instruction (for
+ * suppressing, completing, and terminating exceptions).
+ */
+#define EX_TABLE_INSN(_fault, _target)					\
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_FIXUP, __stringify(%%r0), __stringify(%%r0), 0)
 
-#define EX_TABLE_AMODE31(_fault, _target)				\
+#define EX_TABLE_INSN_AMODE31(_fault, _target)				\
 	__EX_TABLE(.amode31.ex_table, _fault, _target, EX_TYPE_FIXUP, __stringify(%%r0), __stringify(%%r0), 0)
 
-#define EX_TABLE_UA_FAULT(_fault, _target, _regerr)			\
+#define EX_TABLE_INSN_UA_FAULT(_fault, _target, _regerr)		\
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_UA_FAULT, _regerr, _regerr, 0)
 
-#define EX_TABLE_UA_LOAD_REG(_fault, _target, _regerr, _regzero)	\
+#define EX_TABLE_INSN_UA_LOAD_REG(_fault, _target, _regerr, _regzero)	\
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_UA_LOAD_REG, _regerr, _regzero, 0)
 
-#define EX_TABLE_UA_LOAD_REGPAIR(_fault, _target, _regerr, _regzero)	\
+#define EX_TABLE_INSN_UA_LOAD_REGPAIR(_fault, _target, _regerr, _regzero) \
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_UA_LOAD_REGPAIR, _regerr, _regzero, 0)
 
-#define EX_TABLE_ZEROPAD(_fault, _target, _regdata, _regaddr)		\
+#define EX_TABLE_INSN_ZEROPAD(_fault, _target, _regdata, _regaddr)	\
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_ZEROPAD, _regdata, _regaddr, 0)
 
-#define EX_TABLE_FPC(_fault, _target)					\
+#define EX_TABLE_INSN_FPC(_fault, _target)				\
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_FPC, __stringify(%%r0), __stringify(%%r0), 0)
 
-#define EX_TABLE_UA_MVCOS_TO(_fault, _target)				\
+#define EX_TABLE_INSN_UA_MVCOS_TO(_fault, _target)			\
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_UA_MVCOS_TO, __stringify(%%r0), __stringify(%%r0), 0)
 
-#define EX_TABLE_UA_MVCOS_FROM(_fault, _target)				\
+#define EX_TABLE_INSN_UA_MVCOS_FROM(_fault, _target)			\
 	__EX_TABLE(__ex_table, _fault, _target, EX_TYPE_UA_MVCOS_FROM, __stringify(%%r0), __stringify(%%r0), 0)
 
-#define EX_TABLE_INSN				EX_TABLE
-#define EX_TABLE_INSN_AMODE31			EX_TABLE_AMODE31
-#define EX_TABLE_INSN_UA_FAULT			EX_TABLE_UA_FAULT
-#define EX_TABLE_INSN_UA_LOAD_REG		EX_TABLE_UA_LOAD_REG
-#define EX_TABLE_INSN_UA_LOAD_REGPAIR		EX_TABLE_UA_LOAD_REGPAIR
-#define EX_TABLE_INSN_ZEROPAD			EX_TABLE_ZEROPAD
-#define EX_TABLE_INSN_FPC			EX_TABLE_FPC
-#define EX_TABLE_INSN_UA_MVCOS_TO		EX_TABLE_UA_MVCOS_TO
-#define EX_TABLE_INSN_UA_MVCOS_FROM		EX_TABLE_UA_MVCOS_FROM
-
 #endif /* __ASM_EXTABLE_H */
diff --git a/arch/s390/include/asm/extable.h b/arch/s390/include/asm/extable.h
index af6ba52743e9..5032c648781b 100644
--- a/arch/s390/include/asm/extable.h
+++ b/arch/s390/include/asm/extable.h
@@ -5,25 +5,14 @@
 #include <asm/ptrace.h>
 #include <linux/compiler.h>
 
-/*
- * The exception table consists of three addresses:
- *
- * - Address of an instruction that is allowed to fault.
- * - Address at which the program should continue.
- * - Optional address of handler that takes pt_regs * argument and runs in
- *   interrupt context.
- *
- * No registers are modified, so it is entirely up to the continuation code
- * to figure out what to do.
- *
- * All the routines below use bits of fixup code that are out of line
- * with the main instruction path.  This means when everything is well,
- * we don't even have to jump over them.  Further, they do not intrude
- * on our cache or tlb entries.
+/**
+ * struct exception_table_entry - entry in exception table
+ * @insn:  Relative pointer to faulting instruction
+ * @fixup: Relative pointer to instruction where execution resumes
+ * @type:  Exception type
+ * @data:  handler-specific data
  */
-
-struct exception_table_entry
-{
+struct exception_table_entry {
 	int insn, fixup;
 	short type, data;
 };
diff --git a/arch/s390/mm/extable.c b/arch/s390/mm/extable.c
index ca6a1efa49ee..6bed074f1c5f 100644
--- a/arch/s390/mm/extable.c
+++ b/arch/s390/mm/extable.c
@@ -121,8 +121,6 @@ bool fixup_exception(struct pt_regs *regs)
 	const struct exception_table_entry *ex;
 
 	ex = s390_search_extables(get_trap_ip(regs));
-	if (!ex)
-		ex = s390_search_extables(instruction_pointer(regs));
 	if (!ex)
 		return false;
 	switch (ex->type) {
-- 
2.53.0